2024-02-10T06:23:19.955 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T06:23:19.955 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144 --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144/orig.config.yaml 2024-02-10T06:23:20.004 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T06:23:20.094 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144 branch: main description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7555144' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECQFzsaOFTh8GfRTSAuuSnfCajYZ6Z2X8NcWHGxHSyKXA+hxX9LMFhsdLST9AiQQVAF3Y7coUcH19fnM4j0OWg= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFfrJwNnhIEcAMsqbv9JltU8sC0UhFdtsSv12aijf1VN69nMjVNqLoZw2zKY/IQdBcIxLXKRmXHyPAJbN+yqu0= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/yuriw/src/github.com_ceph_ceph-c_main/qa/suites/orch/cephadm/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-10T06:23:20.094 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T06:23:20.095 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T06:23:20.095 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T06:23:20.097 INFO:teuthology.task.internal:Checking packages... 2024-02-10T06:23:20.120 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T06:23:20.120 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:23:20.120 INFO:teuthology.packaging:ref: None 2024-02-10T06:23:20.120 INFO:teuthology.packaging:tag: None 2024-02-10T06:23:20.120 INFO:teuthology.packaging:branch: main 2024-02-10T06:23:20.121 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:23:20.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T06:23:20.362 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T06:23:20.364 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T06:23:20.371 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T06:23:20.371 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T06:23:20.415 INFO:teuthology.task.internal:Saving configuration 2024-02-10T06:23:20.433 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T06:23:20.457 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:23:20.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:18:12.058384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECQFzsaOFTh8GfRTSAuuSnfCajYZ6Z2X8NcWHGxHSyKXA+hxX9LMFhsdLST9AiQQVAF3Y7coUcH19fnM4j0OWg='} 2024-02-10T06:23:20.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:18:12.060291', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFfrJwNnhIEcAMsqbv9JltU8sC0UhFdtsSv12aijf1VN69nMjVNqLoZw2zKY/IQdBcIxLXKRmXHyPAJbN+yqu0='} 2024-02-10T06:23:20.508 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T06:23:20.515 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-10T06:23:20.516 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-10T06:23:20.516 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T06:23:20.618 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd93de44550>, signals=[15]) 2024-02-10T06:23:20.619 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T06:23:20.626 INFO:teuthology.task.internal:Opening connections... 2024-02-10T06:23:20.626 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:23:20.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:23:20.720 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-02-10T06:23:20.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:23:20.808 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T06:23:20.815 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2024-02-10T06:23:20.832 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2024-02-10T06:23:20.832 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2024-02-10T06:23:20.889 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2024-02-10T06:23:20.889 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="9" 2024-02-10T06:23:20.889 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="9" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el9" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:LOGO="fedora-logo-icon" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T06:23:20.890 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T06:23:20.891 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2024-02-10T06:23:20.914 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-02-10T06:23:20.931 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-02-10T06:23:20.931 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-02-10T06:23:20.986 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-02-10T06:23:20.986 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-02-10T06:23:20.986 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-02-10T06:23:20.986 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T06:23:20.987 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T06:23:20.988 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-02-10T06:23:21.013 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T06:23:21.024 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T06:23:21.073 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T06:23:21.073 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2024-02-10T06:23:21.076 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-02-10T06:23:21.092 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T06:23:21.140 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T06:23:21.140 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2024-02-10T06:23:21.142 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-02-10T06:23:21.189 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T06:23:21.262 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T06:23:21.294 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T06:23:21.294 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T06:23:21.295 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T06:23:21.295 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T06:23:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:23:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:23:21.313 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-02-10T06:23:21.313 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-02-10T06:23:21.369 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-361.el9.x86_64 2024-02-10T06:23:21.369 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-361.el9.x86_64 2024-02-10T06:23:21.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-02-10T06:23:21.370 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-361.el9.x86_64 2024-02-10T06:23:21.370 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.14.0-361.el9.x86_64 2024-02-10T06:23:21.370 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2024-02-10T06:23:23.440 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T06:23:23.516 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T06:23:24.875 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-02-10T06:23:25.074 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-10T06:23:25.948 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-10T06:23:29.215 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 19 MB 00:02 2024-02-10T06:23:35.633 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.9 MB 00:02 2024-02-10T06:23:37.000 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 15 kB 00:00 2024-02-10T06:23:38.055 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 638 kB/s | 8.0 MB 00:12 2024-02-10T06:23:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 20 MB 00:02 2024-02-10T06:23:44.809 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-10T06:23:47.116 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T06:23:47.116 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T06:23:47.213 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:23:47.218 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-02-10T06:23:47.219 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:23:47.220 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 99 M 2024-02-10T06:23:47.220 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 122 M 2024-02-10T06:23:47.220 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:23:47.569 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-10T06:23:48.061 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 25 MB/s | 21 MB 00:00 2024-02-10T06:23:48.255 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:09 2024-02-10T06:23:48.736 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 29 MB/s | 33 MB 00:01 2024-02-10T06:23:49.136 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 20 MB/s | 39 MB 00:01 2024-02-10T06:23:49.139 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:23:49.139 INFO:teuthology.orchestra.run.smithi181.stdout:Total 52 MB/s | 99 MB 00:01 2024-02-10T06:23:49.140 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:23:49.511 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:23:49.512 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:23:50.558 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:23:50.559 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:23:52.731 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 5.9 MB 00:00 2024-02-10T06:23:52.989 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:23:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:23:53.778 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:23:54.165 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-02-10T06:23:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2024-02-10T06:23:56.550 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:23:56.739 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:24:00.483 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-10T06:24:01.490 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:24:01.692 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:02.723 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T06:24:02.723 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T06:24:02.823 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:24:02.828 INFO:teuthology.orchestra.run.smithi073.stdout:Install 4 Packages 2024-02-10T06:24:02.829 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:02.829 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 99 M 2024-02-10T06:24:02.829 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 122 M 2024-02-10T06:24:02.829 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:24:03.168 INFO:teuthology.orchestra.run.smithi073.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-02-10T06:24:03.718 INFO:teuthology.orchestra.run.smithi073.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-10T06:24:04.293 INFO:teuthology.orchestra.run.smithi073.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 30 MB/s | 33 MB 00:01 2024-02-10T06:24:04.718 INFO:teuthology.orchestra.run.smithi073.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T06:24:04.721 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:24:04.721 INFO:teuthology.orchestra.run.smithi073.stdout:Total 52 MB/s | 99 MB 00:01 2024-02-10T06:24:04.722 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:24:05.100 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:24:05.100 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:24:06.154 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:24:06.155 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:24:06.442 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:08.547 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:24:09.193 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:24:09.327 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:24:12.079 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:24:12.261 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:24:16.935 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:24:17.164 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:21.880 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:35.600 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:35.600 INFO:teuthology.orchestra.run.smithi181.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T06:24:35.600 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:24:35.672 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:36.051 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:36.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:24:36.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:24:36.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:24:36.446 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:36.446 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:24:36.447 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:24:36.967 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:24:37.098 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:24:37.098 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:24:37.098 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T06:24:37.098 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T06:24:37.098 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T06:24:37.098 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-10T06:24:37.099 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T06:24:37.099 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:24:37.206 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:24:37.207 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:24:37.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-02-10T06:24:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-02-10T06:24:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-02-10T06:24:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-02-10T06:24:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 68.el9 2024-02-10T06:24:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8264254 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-02-10T06:24:37.246 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:24:37.247 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T06:24:37.247 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T06:24:37.247 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T06:24:37.247 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-02-10T06:24:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:24:37.247 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T06:24:37.248 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T06:24:37.248 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-419.el9.x86_64 2024-02-10T06:24:37.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T06:24:37.832 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-02-10T06:24:39.341 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T06:24:39.368 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-02-10T06:24:39.371 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-02-10T06:24:39.390 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.0r0ot4NwUb 2024-02-10T06:24:39.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0r0ot4NwUb 2024-02-10T06:24:39.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.0r0ot4NwUb 2024-02-10T06:24:39.593 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.0r0ot4NwUb is 6KB 2024-02-10T06:24:39.642 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.0r0ot4NwUb 2024-02-10T06:24:39.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T06:24:39.720 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T06:24:39.720 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T06:24:39.720 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T06:24:39.720 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T06:24:39.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T06:24:39.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-02-10T06:24:51.287 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:51.287 INFO:teuthology.orchestra.run.smithi073.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T06:24:51.288 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:51.360 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:51.800 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:51.800 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:24:51.800 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:24:51.800 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:24:52.238 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:24:52.238 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:52.238 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:24:52.239 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T06:24:52.239 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T06:24:52.239 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T06:24:52.239 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T06:24:52.239 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:52.239 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:24:52.757 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:24:52.889 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:24:52.889 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:24:52.889 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T06:24:52.889 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T06:24:52.889 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T06:24:52.889 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-10T06:24:52.889 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T06:24:52.890 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:24:52.996 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:24:52.997 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:24:52.997 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.06 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 68.el9 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Group : Unspecified 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 8264254 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T06:24:53.031 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : builder@centos.org 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : CentOS 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:24:53.032 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T06:24:53.033 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T06:24:53.033 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 5.14.0-419.el9.x86_64 2024-02-10T06:24:53.033 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T06:24:53.640 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2024-02-10T06:24:55.040 INFO:teuthology.orchestra.run.smithi073.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T06:24:55.075 INFO:teuthology.orchestra.run.smithi073.stderr:done 2024-02-10T06:24:55.077 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-02-10T06:24:55.092 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.8rEwgSwMUF 2024-02-10T06:24:55.092 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8rEwgSwMUF 2024-02-10T06:24:55.162 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.8rEwgSwMUF 2024-02-10T06:24:55.305 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.8rEwgSwMUF is 6KB 2024-02-10T06:24:55.353 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.8rEwgSwMUF 2024-02-10T06:24:55.367 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T06:24:55.432 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T06:24:55.433 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T06:24:55.433 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T06:24:55.433 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T06:24:55.434 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T06:24:55.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2024-02-10T06:25:09.876 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-10T06:25:09.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T06:25:09.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:25:25.584 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-10T06:25:25.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-02-10T06:25:25.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:26:09.915 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T06:26:18.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T06:26:18.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:26:22.289 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-02-10T06:26:22.720 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T06:26:22.720 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T06:26:22.721 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-02-10T06:26:22.735 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-419.el9.x86_64 2024-02-10T06:26:22.735 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T06:26:22.735 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T06:26:22.736 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T06:26:23.736 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-10T06:26:23.737 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T06:26:23.805 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-02-10T06:26:23.838 DEBUG:teuthology.parallel:result is None 2024-02-10T06:26:25.588 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T06:26:34.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-02-10T06:26:34.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:26:34.975 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-02-10T06:26:35.406 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-02-10T06:26:35.406 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T06:26:35.406 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-02-10T06:26:35.420 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-419.el9.x86_64 2024-02-10T06:26:35.420 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T06:26:35.420 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T06:26:35.420 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T06:26:36.421 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-10T06:26:36.421 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T06:26:36.487 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2024-02-10T06:26:36.522 DEBUG:teuthology.parallel:result is None 2024-02-10T06:26:36.523 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T06:26:36.533 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T06:26:36.534 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T06:26:36.535 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T06:26:36.555 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T06:26:36.574 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T06:26:36.624 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T06:26:36.624 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T06:26:36.626 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T06:26:36.727 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T06:26:36.734 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T06:26:36.734 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T06:26:36.736 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T06:26:36.769 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:26:36.782 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:26:36.783 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:26:36.795 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:26:36.797 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T06:26:36.806 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T06:26:36.806 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T06:26:36.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T06:26:36.855 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T06:26:36.868 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T06:26:36.869 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T06:26:36.881 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T06:26:36.911 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:26:36.981 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:26:37.033 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:26:37.120 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:26:37.166 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:26:37.166 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T06:26:37.238 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:26:37.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:26:37.344 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:26:37.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:26:37.432 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:26:37.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T06:26:37.501 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2024-02-10T06:26:37.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-02-10T06:26:37.555 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:26:37.589 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:26:37.848 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T06:26:37.857 INFO:teuthology.task.internal:Starting timer... 2024-02-10T06:26:37.857 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T06:26:37.902 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T06:26:37.935 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T06:26:37.935 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2024-02-10T06:26:37.981 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2024-02-10T06:26:37.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-02-10T06:26:38.041 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-02-10T06:26:38.043 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T06:26:38.043 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2024-02-10T06:26:38.071 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2024-02-10T06:26:38.072 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-02-10T06:26:38.091 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-02-10T06:26:38.092 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi073.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-02-10T06:26:38.092 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:26:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:26:38.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:26:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:26:38.170 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T06:26:38.170 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2024-02-10T06:26:38.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-02-10T06:26:38.260 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T06:26:38.270 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T06:26:38.289 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T06:26:38.290 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi073.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T06:30:47.535 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi073.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-02-10T06:30:47.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-02-10T06:30:47.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:47.627 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-02-10T06:30:47.702 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-02-10T06:30:47.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T06:30:47.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:47.790 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-02-10T06:30:47.867 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T06:30:47.867 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T06:30:47.878 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T06:30:47.878 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T06:30:47.878 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:30:47.880 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T06:30:47.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:30:47.908 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T06:30:47.924 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T06:30:47.939 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T06:30:47.953 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2024-02-10T06:30:47.956 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T06:30:47.966 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2024-02-10T06:30:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T06:30:47.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-02-10T06:30:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T06:30:48.000 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-02-10T06:30:48.016 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T06:30:48.032 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T06:30:48.048 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:30:48.052 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:30:48.052 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-02-10T06:30:48.052 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.052 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.052 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.052 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.082 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:30:48.085 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:30:48.085 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-02-10T06:30:48.085 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.086 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.086 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.086 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:30:48.087 INFO:teuthology.run_tasks:Running task install... 2024-02-10T06:30:48.095 DEBUG:teuthology.task.install:project ceph 2024-02-10T06:30:48.096 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-10T06:30:48.096 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T06:30:48.096 INFO:teuthology.task.install:Using flavor: default 2024-02-10T06:30:48.102 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T06:30:48.102 INFO:teuthology.task.install:extra packages: [] 2024-02-10T06:30:48.103 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T06:30:48.103 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:30:48.103 INFO:teuthology.packaging:ref: None 2024-02-10T06:30:48.103 INFO:teuthology.packaging:tag: None 2024-02-10T06:30:48.103 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:30:48.103 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:30:48.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:30:48.106 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T06:30:48.106 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:30:48.106 INFO:teuthology.packaging:ref: None 2024-02-10T06:30:48.106 INFO:teuthology.packaging:tag: None 2024-02-10T06:30:48.106 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:30:48.106 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:30:48.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:30:48.333 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/ 2024-02-10T06:30:48.334 INFO:teuthology.task.install.rpm:Package version is 17.2.7-505.g72ce7c1d 2024-02-10T06:30:48.348 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/ 2024-02-10T06:30:48.349 INFO:teuthology.task.install.rpm:Package version is 17.2.7-505.g72ce7c1d 2024-02-10T06:30:48.483 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T06:30:48.484 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:30:48.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T06:30:48.486 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T06:30:48.487 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:30:48.487 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T06:30:48.514 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T06:30:48.514 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:30:48.514 INFO:teuthology.packaging:ref: None 2024-02-10T06:30:48.514 INFO:teuthology.packaging:tag: None 2024-02-10T06:30:48.514 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:30:48.514 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:30:48.514 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T06:30:48.516 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T06:30:48.516 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:30:48.516 INFO:teuthology.packaging:ref: None 2024-02-10T06:30:48.516 INFO:teuthology.packaging:tag: None 2024-02-10T06:30:48.516 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:30:48.516 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:30:48.516 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T06:30:48.582 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T06:30:48.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T06:30:48.662 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T06:30:48.665 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T06:30:48.728 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2024-02-10T06:30:48.730 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2024-02-10T06:30:48.731 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-02-10T06:30:48.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-02-10T06:30:48.973 INFO:teuthology.orchestra.run.smithi181.stdout:74 files removed 2024-02-10T06:30:48.984 INFO:teuthology.orchestra.run.smithi073.stdout:74 files removed 2024-02-10T06:30:48.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-02-10T06:30:49.013 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw 2024-02-10T06:30:49.663 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 225 kB/s | 77 kB 00:00 2024-02-10T06:30:49.681 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 219 kB/s | 77 kB 00:00 2024-02-10T06:30:50.030 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-02-10T06:30:50.040 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-02-10T06:30:50.366 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 6.2 kB/s | 1.8 kB 00:00 2024-02-10T06:30:50.380 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-02-10T06:30:50.943 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-10T06:30:50.983 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-10T06:30:52.436 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-02-10T06:30:52.600 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-02-10T06:30:53.507 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-02-10T06:30:53.528 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-02-10T06:30:56.825 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 19 MB 00:02 2024-02-10T06:30:57.244 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-02-10T06:31:00.926 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 5.9 MB 00:00 2024-02-10T06:31:01.203 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 13 MB/s | 5.9 MB 00:00 2024-02-10T06:31:02.377 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2024-02-10T06:31:02.728 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2024-02-10T06:31:03.220 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 29 MB/s | 20 MB 00:00 2024-02-10T06:31:04.796 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-10T06:31:08.187 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-02-10T06:31:09.713 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-10T06:31:10.855 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:31:10.858 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 14 M 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.1 M 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 5.0 M 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 23 M 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:31:10.859 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 635 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 458 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 4.9 M 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 45 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 116 k 2024-02-10T06:31:10.860 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 156 k 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 320 k 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 293 k 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 99 k 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout:Install 22 Packages 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-02-10T06:31:10.861 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:10.862 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 58 M 2024-02-10T06:31:10.862 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:31:11.954 INFO:teuthology.orchestra.run.smithi181.stdout:(1/24): ceph-base-17.2.7-505.g72ce7c1d.el9.x86_ 5.5 MB/s | 5.0 MB 00:00 2024-02-10T06:31:12.004 INFO:teuthology.orchestra.run.smithi181.stdout:(2/24): ceph-selinux-17.2.7-505.g72ce7c1d.el9.x 504 kB/s | 25 kB 00:00 2024-02-10T06:31:12.122 INFO:teuthology.orchestra.run.smithi181.stdout:(3/24): libcephfs2-17.2.7-505.g72ce7c1d.el9.x86 5.3 MB/s | 635 kB 00:00 2024-02-10T06:31:12.238 INFO:teuthology.orchestra.run.smithi181.stdout:(4/24): libradosstriper1-17.2.7-505.g72ce7c1d.e 3.8 MB/s | 458 kB 00:00 2024-02-10T06:31:12.374 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:31:12.376 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 14 M 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.1 M 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 5.0 M 2024-02-10T06:31:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 23 M 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 635 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 458 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 4.9 M 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 45 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 116 k 2024-02-10T06:31:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 156 k 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 320 k 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 293 k 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 99 k 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout:Install 22 Packages 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 2 Packages 2024-02-10T06:31:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:12.380 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 58 M 2024-02-10T06:31:12.380 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:31:12.689 INFO:teuthology.orchestra.run.smithi181.stdout:(5/24): ceph-radosgw-17.2.7-505.g72ce7c1d.el9.x 8.5 MB/s | 14 MB 00:01 2024-02-10T06:31:12.781 INFO:teuthology.orchestra.run.smithi181.stdout:(6/24): python3-ceph-argparse-17.2.7-505.g72ce7 493 kB/s | 45 kB 00:00 2024-02-10T06:31:12.873 INFO:teuthology.orchestra.run.smithi181.stdout:(7/24): python3-ceph-common-17.2.7-505.g72ce7c1 1.2 MB/s | 116 kB 00:00 2024-02-10T06:31:12.981 INFO:teuthology.orchestra.run.smithi181.stdout:(8/24): librgw2-17.2.7-505.g72ce7c1d.el9.x86_64 6.6 MB/s | 4.9 MB 00:00 2024-02-10T06:31:13.006 INFO:teuthology.orchestra.run.smithi181.stdout:(9/24): python3-cephfs-17.2.7-505.g72ce7c1d.el9 1.1 MB/s | 156 kB 00:00 2024-02-10T06:31:13.074 INFO:teuthology.orchestra.run.smithi181.stdout:(10/24): python3-rados-17.2.7-505.g72ce7c1d.el9 3.4 MB/s | 320 kB 00:00 2024-02-10T06:31:13.108 INFO:teuthology.orchestra.run.smithi181.stdout:(11/24): python3-rbd-17.2.7-505.g72ce7c1d.el9.x 2.8 MB/s | 293 kB 00:00 2024-02-10T06:31:13.166 INFO:teuthology.orchestra.run.smithi181.stdout:(12/24): python3-rgw-17.2.7-505.g72ce7c1d.el9.x 1.1 MB/s | 99 kB 00:00 2024-02-10T06:31:13.233 INFO:teuthology.orchestra.run.smithi181.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-02-10T06:31:13.258 INFO:teuthology.orchestra.run.smithi181.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 486 kB/s | 44 kB 00:00 2024-02-10T06:31:13.458 INFO:teuthology.orchestra.run.smithi181.stdout:(15/24): ceph-common-17.2.7-505.g72ce7c1d.el9.x 9.3 MB/s | 23 MB 00:02 2024-02-10T06:31:13.484 INFO:teuthology.orchestra.run.smithi181.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 661 kB 00:00 2024-02-10T06:31:13.509 INFO:teuthology.orchestra.run.smithi181.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-02-10T06:31:13.534 INFO:teuthology.orchestra.run.smithi181.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 552 kB/s | 41 kB 00:00 2024-02-10T06:31:13.639 INFO:teuthology.orchestra.run.smithi073.stdout:(1/24): ceph-base-17.2.7-505.g72ce7c1d.el9.x86_ 4.1 MB/s | 5.0 MB 00:01 2024-02-10T06:31:13.706 INFO:teuthology.orchestra.run.smithi073.stdout:(2/24): ceph-selinux-17.2.7-505.g72ce7c1d.el9.x 378 kB/s | 25 kB 00:00 2024-02-10T06:31:13.814 INFO:teuthology.orchestra.run.smithi073.stdout:(3/24): libcephfs2-17.2.7-505.g72ce7c1d.el9.x86 5.7 MB/s | 635 kB 00:00 2024-02-10T06:31:13.932 INFO:teuthology.orchestra.run.smithi073.stdout:(4/24): libradosstriper1-17.2.7-505.g72ce7c1d.e 3.8 MB/s | 458 kB 00:00 2024-02-10T06:31:14.143 INFO:teuthology.orchestra.run.smithi181.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 77 kB/s | 49 kB 00:00 2024-02-10T06:31:14.168 INFO:teuthology.orchestra.run.smithi181.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 106 kB/s | 67 kB 00:00 2024-02-10T06:31:14.565 INFO:teuthology.orchestra.run.smithi073.stdout:(5/24): ceph-radosgw-17.2.7-505.g72ce7c1d.el9.x 6.6 MB/s | 14 MB 00:02 2024-02-10T06:31:14.657 INFO:teuthology.orchestra.run.smithi073.stdout:(6/24): librgw2-17.2.7-505.g72ce7c1d.el9.x86_64 6.8 MB/s | 4.9 MB 00:00 2024-02-10T06:31:14.682 INFO:teuthology.orchestra.run.smithi073.stdout:(7/24): python3-ceph-argparse-17.2.7-505.g72ce7 386 kB/s | 45 kB 00:00 2024-02-10T06:31:14.710 INFO:teuthology.orchestra.run.smithi181.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 257 kB/s | 309 kB 00:01 2024-02-10T06:31:14.741 INFO:teuthology.orchestra.run.smithi073.stdout:(8/24): python3-ceph-common-17.2.7-505.g72ce7c1 1.4 MB/s | 116 kB 00:00 2024-02-10T06:31:14.766 INFO:teuthology.orchestra.run.smithi073.stdout:(9/24): python3-cephfs-17.2.7-505.g72ce7c1d.el9 1.8 MB/s | 156 kB 00:00 2024-02-10T06:31:14.825 INFO:teuthology.orchestra.run.smithi073.stdout:(10/24): python3-rados-17.2.7-505.g72ce7c1d.el9 3.8 MB/s | 320 kB 00:00 2024-02-10T06:31:14.858 INFO:teuthology.orchestra.run.smithi073.stdout:(11/24): python3-rbd-17.2.7-505.g72ce7c1d.el9.x 3.1 MB/s | 293 kB 00:00 2024-02-10T06:31:14.917 INFO:teuthology.orchestra.run.smithi073.stdout:(12/24): python3-rgw-17.2.7-505.g72ce7c1d.el9.x 1.1 MB/s | 99 kB 00:00 2024-02-10T06:31:15.035 INFO:teuthology.orchestra.run.smithi181.stdout:(22/24): librados2-17.2.7-505.g72ce7c1d.el9.x86 3.5 MB/s | 3.1 MB 00:00 2024-02-10T06:31:15.143 INFO:teuthology.orchestra.run.smithi073.stdout:(13/24): ceph-common-17.2.7-505.g72ce7c1d.el9.x 8.3 MB/s | 23 MB 00:02 2024-02-10T06:31:15.168 INFO:teuthology.orchestra.run.smithi073.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 177 kB/s | 44 kB 00:00 2024-02-10T06:31:15.185 INFO:teuthology.orchestra.run.smithi073.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 488 kB/s | 159 kB 00:00 2024-02-10T06:31:15.218 INFO:teuthology.orchestra.run.smithi073.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-02-10T06:31:15.243 INFO:teuthology.orchestra.run.smithi073.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-02-10T06:31:15.268 INFO:teuthology.orchestra.run.smithi073.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 495 kB/s | 41 kB 00:00 2024-02-10T06:31:15.318 INFO:teuthology.orchestra.run.smithi181.stdout:(23/24): librbd1-17.2.7-505.g72ce7c1d.el9.x86_6 4.8 MB/s | 2.9 MB 00:00 2024-02-10T06:31:15.827 INFO:teuthology.orchestra.run.smithi073.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 87 kB/s | 49 kB 00:00 2024-02-10T06:31:15.844 INFO:teuthology.orchestra.run.smithi073.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 117 kB/s | 67 kB 00:00 2024-02-10T06:31:16.086 INFO:teuthology.orchestra.run.smithi073.stdout:(21/24): librados2-17.2.7-505.g72ce7c1d.el9.x86 13 MB/s | 3.1 MB 00:00 2024-02-10T06:31:16.245 INFO:teuthology.orchestra.run.smithi073.stdout:(22/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 301 kB/s | 309 kB 00:01 2024-02-10T06:31:16.411 INFO:teuthology.orchestra.run.smithi073.stdout:(23/24): librbd1-17.2.7-505.g72ce7c1d.el9.x86_6 9.0 MB/s | 2.9 MB 00:00 2024-02-10T06:31:17.493 INFO:teuthology.orchestra.run.smithi181.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 483 kB/s | 1.6 MB 00:03 2024-02-10T06:31:17.500 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:31:17.500 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.7 MB/s | 58 MB 00:06 2024-02-10T06:31:17.500 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:31:17.533 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:31:17.534 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:31:18.001 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:31:18.001 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:31:18.862 INFO:teuthology.orchestra.run.smithi073.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 534 kB/s | 1.6 MB 00:03 2024-02-10T06:31:18.868 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:31:18.868 INFO:teuthology.orchestra.run.smithi073.stdout:Total 8.9 MB/s | 58 MB 00:06 2024-02-10T06:31:18.868 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:31:18.901 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:31:18.901 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:31:19.256 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:31:19.369 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:31:19.370 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:31:19.602 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-10T06:31:19.662 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:31:19.754 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:31:19.840 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:31:19.889 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:31:19.977 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:31:20.030 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-10T06:31:20.068 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-10T06:31:20.502 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-10T06:31:20.542 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:31:20.604 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:31:20.608 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:31:20.633 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-10T06:31:20.676 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 10/26 2024-02-10T06:31:20.716 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:31:20.782 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:31:20.803 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:31:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:31:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-10T06:31:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-10T06:31:20.942 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-10T06:31:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-10T06:31:21.005 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:31:21.104 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:31:21.205 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:31:21.254 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-10T06:31:21.256 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:31:21.290 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:31:21.337 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:31:21.363 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:31:21.383 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-10T06:31:21.412 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-10T06:31:21.436 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 19/26 2024-02-10T06:31:21.467 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 20/26 2024-02-10T06:31:21.846 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-10T06:31:21.876 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:31:21.939 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:31:21.968 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-10T06:31:22.010 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 10/26 2024-02-10T06:31:22.056 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:31:22.125 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:31:22.155 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:31:22.203 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:31:22.243 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-10T06:31:22.278 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-10T06:31:22.311 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-10T06:31:22.606 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-10T06:31:22.635 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:31:22.723 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:31:22.797 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 19/26 2024-02-10T06:31:22.821 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 20/26 2024-02-10T06:31:23.891 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:31:23.931 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:31:24.471 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:31:24.509 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:31:24.600 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:31:24.600 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T06:31:24.600 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T06:31:24.601 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:24.635 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:31:25.265 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:31:25.309 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:31:25.884 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:31:25.921 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:31:26.012 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:31:26.013 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T06:31:26.013 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T06:31:26.013 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:26.039 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-02-10T06:31:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:35.318 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:31:35.318 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /root 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2024-02-10T06:31:35.319 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:35.428 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:31:35.507 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:31:35.507 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:31:35.507 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T06:31:35.507 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:31:35.507 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:31:35.507 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:35.508 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:31:35.606 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:31:35.606 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:31:36.632 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:31:36.693 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:31:36.694 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:31:36.694 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T06:31:36.694 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:31:36.694 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:31:36.694 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:36.695 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:31:36.785 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:31:36.786 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:31:36.886 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:31:36.886 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/26 2024-02-10T06:31:36.886 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:31:36.886 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:31:36.886 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 7/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 8/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 9/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 10/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-10T06:31:36.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-10T06:31:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-10T06:31:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-10T06:31:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-10T06:31:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:31:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-10T06:31:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 25/26 2024-02-10T06:31:37.754 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T06:31:37.755 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.756 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.757 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:37.757 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T06:31:37.757 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:37.757 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:31:38.094 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:31:38.094 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/26 2024-02-10T06:31:38.094 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:31:38.094 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:31:38.094 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 7/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 8/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 9/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 10/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-10T06:31:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-10T06:31:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 25/26 2024-02-10T06:31:38.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-02-10T06:31:38.676 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:30 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:31:38.833 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 48 M 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 162 k 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T06:31:38.834 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 49 M 2024-02-10T06:31:38.835 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 196 M 2024-02-10T06:31:38.836 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:31:39.052 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-02-10T06:31:39.144 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 683 kB/s | 63 kB 00:00 2024-02-10T06:31:39.202 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-02-10T06:31:39.220 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T06:31:39.221 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T06:31:39.222 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.223 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.223 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:39.223 INFO:teuthology.orchestra.run.smithi073.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T06:31:39.223 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:39.223 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:31:39.227 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): libcephsqlite-17.2.7-505.g72ce7c1d.el9.x 415 kB/s | 162 kB 00:00 2024-02-10T06:31:39.269 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-02-10T06:31:39.613 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-test 2024-02-10T06:31:40.188 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:31 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:31:40.346 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 48 M 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T06:31:40.347 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 162 k 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2024-02-10T06:31:40.348 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:40.349 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 49 M 2024-02-10T06:31:40.349 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 196 M 2024-02-10T06:31:40.349 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:31:40.660 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 968 kB/s | 300 kB 00:00 2024-02-10T06:31:40.701 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-02-10T06:31:40.735 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): libcephsqlite-17.2.7-505.g72ce7c1d.el9.x 421 kB/s | 162 kB 00:00 2024-02-10T06:31:40.768 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-02-10T06:31:40.802 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-02-10T06:31:44.727 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): ceph-test-17.2.7-505.g72ce7c1d.el9.x86_6 8.1 MB/s | 48 MB 00:05 2024-02-10T06:31:44.730 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:31:44.730 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.3 MB/s | 49 MB 00:05 2024-02-10T06:31:44.730 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:31:44.747 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:31:44.747 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:31:45.152 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:31:45.152 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:31:45.560 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): ceph-test-17.2.7-505.g72ce7c1d.el9.x86_6 9.2 MB/s | 48 MB 00:05 2024-02-10T06:31:45.563 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:31:45.563 INFO:teuthology.orchestra.run.smithi073.stdout:Total 9.4 MB/s | 49 MB 00:05 2024-02-10T06:31:45.563 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:31:45.580 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:31:45.580 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:31:45.748 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:31:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T06:31:45.853 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T06:31:45.896 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T06:31:45.936 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T06:31:45.964 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:31:45.982 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:31:45.983 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:31:46.566 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:31:46.638 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T06:31:46.677 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T06:31:46.721 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T06:31:46.761 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T06:31:46.789 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:31:50.389 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:31:50.482 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:31:51.213 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:31:51.300 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:31:52.747 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:31:52.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/6 2024-02-10T06:31:52.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/6 2024-02-10T06:31:52.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T06:31:52.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T06:31:52.748 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T06:31:53.138 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T06:31:53.138 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:53.138 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:31:53.138 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:53.139 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-16.el9.x86_64 2024-02-10T06:31:53.139 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:53.139 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T06:31:53.139 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T06:31:53.139 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T06:31:53.139 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:53.140 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:31:53.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-02-10T06:31:53.526 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:31:53.526 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/6 2024-02-10T06:31:53.526 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/6 2024-02-10T06:31:53.526 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T06:31:53.526 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T06:31:53.526 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T06:31:53.910 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T06:31:53.910 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:53.910 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:31:53.910 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:53.910 INFO:teuthology.orchestra.run.smithi073.stdout: jq-1.6-16.el9.x86_64 2024-02-10T06:31:53.911 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:31:53.911 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T06:31:53.911 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T06:31:53.911 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T06:31:53.911 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:53.911 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:31:54.011 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:46 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:31:54.175 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 6.5 k 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:31:54.177 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.1 M 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 1.4 M 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 239 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.7 M 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 15 M 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T06:31:54.178 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T06:31:54.179 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout:Install 33 Packages 2024-02-10T06:31:54.180 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:31:54.181 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 27 M 2024-02-10T06:31:54.181 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 99 M 2024-02-10T06:31:54.181 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:31:54.208 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph 2024-02-10T06:31:54.734 INFO:teuthology.orchestra.run.smithi181.stdout:(1/33): ceph-17.2.7-505.g72ce7c1d.el9.x86_64.rp 36 kB/s | 6.5 kB 00:00 2024-02-10T06:31:54.810 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:45 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:31:54.976 INFO:teuthology.orchestra.run.smithi181.stdout:(2/33): ceph-mgr-17.2.7-505.g72ce7c1d.el9.x86_6 3.3 MB/s | 1.4 MB 00:00 2024-02-10T06:31:54.987 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:31:54.989 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 6.5 k 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.1 M 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 1.4 M 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 239 k 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.7 M 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 15 M 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T06:31:54.990 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T06:31:54.991 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2024-02-10T06:31:54.992 INFO:teuthology.orchestra.run.smithi073.stdout:Install 33 Packages 2024-02-10T06:31:54.993 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:31:54.993 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 27 M 2024-02-10T06:31:54.993 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 99 M 2024-02-10T06:31:54.993 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:31:55.085 INFO:teuthology.orchestra.run.smithi181.stdout:(3/33): ceph-mds-17.2.7-505.g72ce7c1d.el9.x86_6 3.9 MB/s | 2.1 MB 00:00 2024-02-10T06:31:55.160 INFO:teuthology.orchestra.run.smithi181.stdout:(4/33): ceph-mgr-modules-core-17.2.7-505.g72ce7 3.1 MB/s | 239 kB 00:00 2024-02-10T06:31:55.377 INFO:teuthology.orchestra.run.smithi181.stdout:(5/33): ceph-mon-17.2.7-505.g72ce7c1d.el9.x86_6 5.7 MB/s | 3.7 MB 00:00 2024-02-10T06:31:55.402 INFO:teuthology.orchestra.run.smithi181.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 597 kB/s | 144 kB 00:00 2024-02-10T06:31:55.419 INFO:teuthology.orchestra.run.smithi181.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 755 kB/s | 32 kB 00:00 2024-02-10T06:31:55.436 INFO:teuthology.orchestra.run.smithi181.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-10T06:31:55.443 INFO:teuthology.orchestra.run.smithi073.stdout:(1/33): ceph-17.2.7-505.g72ce7c1d.el9.x86_64.rp 26 kB/s | 6.5 kB 00:00 2024-02-10T06:31:55.453 INFO:teuthology.orchestra.run.smithi181.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-10T06:31:55.503 INFO:teuthology.orchestra.run.smithi181.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-02-10T06:31:55.604 INFO:teuthology.orchestra.run.smithi181.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-02-10T06:31:55.629 INFO:teuthology.orchestra.run.smithi181.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 823 kB/s | 103 kB 00:00 2024-02-10T06:31:55.671 INFO:teuthology.orchestra.run.smithi181.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-02-10T06:31:55.696 INFO:teuthology.orchestra.run.smithi181.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 1.7 MB/s | 113 kB 00:00 2024-02-10T06:31:55.721 INFO:teuthology.orchestra.run.smithi181.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-10T06:31:55.952 INFO:teuthology.orchestra.run.smithi073.stdout:(2/33): ceph-mgr-17.2.7-505.g72ce7c1d.el9.x86_6 1.9 MB/s | 1.4 MB 00:00 2024-02-10T06:31:56.064 INFO:teuthology.orchestra.run.smithi181.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 118 kB/s | 43 kB 00:00 2024-02-10T06:31:56.127 INFO:teuthology.orchestra.run.smithi073.stdout:(3/33): ceph-mds-17.2.7-505.g72ce7c1d.el9.x86_6 2.2 MB/s | 2.1 MB 00:00 2024-02-10T06:31:56.219 INFO:teuthology.orchestra.run.smithi073.stdout:(4/33): ceph-mgr-modules-core-17.2.7-505.g72ce7 2.5 MB/s | 239 kB 00:00 2024-02-10T06:31:56.305 INFO:teuthology.orchestra.run.smithi181.stdout:(17/33): ceph-osd-17.2.7-505.g72ce7c1d.el9.x86_ 12 MB/s | 15 MB 00:01 2024-02-10T06:31:56.327 INFO:teuthology.orchestra.run.smithi073.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-02-10T06:31:56.330 INFO:teuthology.orchestra.run.smithi181.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 282 kB/s | 172 kB 00:00 2024-02-10T06:31:56.362 INFO:teuthology.orchestra.run.smithi073.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 936 kB/s | 32 kB 00:00 2024-02-10T06:31:56.389 INFO:teuthology.orchestra.run.smithi181.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-02-10T06:31:56.395 INFO:teuthology.orchestra.run.smithi073.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-10T06:31:56.429 INFO:teuthology.orchestra.run.smithi073.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-10T06:31:56.471 INFO:teuthology.orchestra.run.smithi073.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-02-10T06:31:56.482 INFO:teuthology.orchestra.run.smithi181.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-02-10T06:31:56.538 INFO:teuthology.orchestra.run.smithi073.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-02-10T06:31:56.565 INFO:teuthology.orchestra.run.smithi181.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 76 kB/s | 18 kB 00:00 2024-02-10T06:31:56.579 INFO:teuthology.orchestra.run.smithi073.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-02-10T06:31:56.581 INFO:teuthology.orchestra.run.smithi181.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-02-10T06:31:56.613 INFO:teuthology.orchestra.run.smithi073.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-02-10T06:31:56.655 INFO:teuthology.orchestra.run.smithi073.stdout:(13/33): python3-requests-2.25.1-8.el9.noarch.r 2.7 MB/s | 113 kB 00:00 2024-02-10T06:31:56.657 INFO:teuthology.orchestra.run.smithi181.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-02-10T06:31:56.682 INFO:teuthology.orchestra.run.smithi181.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-02-10T06:31:56.697 INFO:teuthology.orchestra.run.smithi073.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-02-10T06:31:56.708 INFO:teuthology.orchestra.run.smithi181.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 557 kB/s | 358 kB 00:00 2024-02-10T06:31:56.816 INFO:teuthology.orchestra.run.smithi181.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-02-10T06:31:56.822 INFO:teuthology.orchestra.run.smithi073.stdout:(15/33): ceph-mon-17.2.7-505.g72ce7c1d.el9.x86_ 2.7 MB/s | 3.7 MB 00:01 2024-02-10T06:31:56.848 INFO:teuthology.orchestra.run.smithi073.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-02-10T06:31:56.849 INFO:teuthology.orchestra.run.smithi181.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 410 kB/s | 79 kB 00:00 2024-02-10T06:31:56.925 INFO:teuthology.orchestra.run.smithi181.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 827 kB/s | 90 kB 00:00 2024-02-10T06:31:56.959 INFO:teuthology.orchestra.run.smithi181.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-02-10T06:31:56.998 INFO:teuthology.orchestra.run.smithi073.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-02-10T06:31:57.023 INFO:teuthology.orchestra.run.smithi073.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 857 kB/s | 172 kB 00:00 2024-02-10T06:31:57.025 INFO:teuthology.orchestra.run.smithi181.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-02-10T06:31:57.049 INFO:teuthology.orchestra.run.smithi073.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-02-10T06:31:57.051 INFO:teuthology.orchestra.run.smithi181.stdout:(31/33): python3-pecan-1.4.2-3.el9.noarch.rpm 739 kB/s | 272 kB 00:00 2024-02-10T06:31:57.074 INFO:teuthology.orchestra.run.smithi073.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-02-10T06:31:57.099 INFO:teuthology.orchestra.run.smithi073.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-02-10T06:31:57.118 INFO:teuthology.orchestra.run.smithi181.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-02-10T06:31:57.124 INFO:teuthology.orchestra.run.smithi073.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-02-10T06:31:57.150 INFO:teuthology.orchestra.run.smithi073.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-02-10T06:31:57.200 INFO:teuthology.orchestra.run.smithi073.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-02-10T06:31:57.217 INFO:teuthology.orchestra.run.smithi073.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-02-10T06:31:57.267 INFO:teuthology.orchestra.run.smithi073.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-02-10T06:31:57.292 INFO:teuthology.orchestra.run.smithi073.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-02-10T06:31:57.309 INFO:teuthology.orchestra.run.smithi073.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-02-10T06:31:57.343 INFO:teuthology.orchestra.run.smithi073.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-02-10T06:31:57.360 INFO:teuthology.orchestra.run.smithi181.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.0 MB/s | 427 kB 00:00 2024-02-10T06:31:57.363 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:31:57.363 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.4 MB/s | 27 MB 00:03 2024-02-10T06:31:57.364 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:31:57.393 INFO:teuthology.orchestra.run.smithi073.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-02-10T06:31:57.402 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:31:57.402 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:31:57.419 INFO:teuthology.orchestra.run.smithi073.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-02-10T06:31:57.436 INFO:teuthology.orchestra.run.smithi073.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 477 kB/s | 20 kB 00:00 2024-02-10T06:31:57.594 INFO:teuthology.orchestra.run.smithi073.stdout:(33/33): ceph-osd-17.2.7-505.g72ce7c1d.el9.x86_ 9.3 MB/s | 15 MB 00:01 2024-02-10T06:31:57.597 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:31:57.597 INFO:teuthology.orchestra.run.smithi073.stdout:Total 10 MB/s | 27 MB 00:02 2024-02-10T06:31:57.598 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:31:57.634 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:31:57.634 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:31:57.734 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:31:57.734 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:31:57.978 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:31:57.978 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:31:58.995 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:31:59.071 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-10T06:31:59.103 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-10T06:31:59.153 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-10T06:31:59.205 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:31:59.244 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-10T06:31:59.256 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-10T06:31:59.288 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-10T06:31:59.320 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-10T06:31:59.321 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-10T06:31:59.363 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-10T06:31:59.412 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-10T06:31:59.434 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-10T06:31:59.481 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-10T06:31:59.489 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-10T06:31:59.523 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-10T06:31:59.536 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-10T06:31:59.584 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-10T06:31:59.594 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-10T06:31:59.645 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-10T06:31:59.648 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-10T06:31:59.687 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-10T06:31:59.735 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-10T06:31:59.795 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-10T06:31:59.832 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-10T06:31:59.874 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-10T06:31:59.904 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-10T06:31:59.939 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-10T06:31:59.968 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-10T06:31:59.981 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-10T06:32:00.034 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-10T06:32:00.043 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-10T06:32:00.056 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-10T06:32:00.078 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:32:00.091 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-10T06:32:00.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-10T06:32:00.120 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-10T06:32:00.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-10T06:32:00.187 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-10T06:32:00.196 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-10T06:32:00.222 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-10T06:32:00.231 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:32:00.265 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-10T06:32:00.305 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-10T06:32:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-10T06:32:00.340 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-10T06:32:00.353 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-10T06:32:00.374 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-10T06:32:00.424 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-10T06:32:00.470 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-10T06:32:00.505 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-10T06:32:00.560 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-10T06:32:00.586 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-10T06:32:00.677 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:32:00.722 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-10T06:32:00.837 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:32:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 28/33 2024-02-10T06:32:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:32:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:32:00.892 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:00.892 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T06:32:00.892 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:32:00.892 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:32:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:00.988 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 28/33 2024-02-10T06:32:01.019 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:32:01.044 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:32:01.045 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:01.045 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T06:32:01.045 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:32:01.045 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:32:01.045 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:02.225 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:32:02.252 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:32:02.252 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:02.252 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T06:32:02.252 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:32:02.252 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:32:02.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:02.381 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:32:02.406 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:32:02.406 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:02.407 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T06:32:02.407 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:32:02.407 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:32:02.407 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:02.626 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:32:02.649 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:32:02.649 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:02.649 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T06:32:02.649 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:32:02.649 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:32:02.649 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:02.790 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:32:02.812 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:32:02.812 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:02.812 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T06:32:02.813 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:32:02.813 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:32:02.813 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:02.852 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:32:02.875 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:32:02.875 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:02.875 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T06:32:02.875 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:32:02.875 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:32:02.875 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:02.941 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:32:03.015 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:32:03.038 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:32:03.038 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:03.038 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T06:32:03.038 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:32:03.038 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:32:03.038 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:03.104 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:32:04.682 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:32:04.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 6/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-10T06:32:04.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-10T06:32:04.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-10T06:32:04.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-10T06:32:04.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-10T06:32:04.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:32:04.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-10T06:32:04.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-10T06:32:04.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-10T06:32:04.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-10T06:32:04.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-10T06:32:04.788 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:32:04.788 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/33 2024-02-10T06:32:04.788 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/33 2024-02-10T06:32:04.788 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 6/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-10T06:32:04.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-10T06:32:04.790 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-10T06:32:04.791 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-10T06:32:04.791 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:32:04.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-10T06:32:04.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-10T06:32:04.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-10T06:32:04.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-10T06:32:04.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-10T06:32:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-10T06:32:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T06:32:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T06:32:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:05.531 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T06:32:05.600 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T06:32:05.601 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T06:32:05.602 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T06:32:05.603 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T06:32:05.603 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T06:32:05.603 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:05.603 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:05.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-02-10T06:32:05.961 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-base 2024-02-10T06:32:06.368 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:58 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:06.476 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:06.527 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:06.528 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:06.528 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:06.537 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:57 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:06.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-02-10T06:32:06.644 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:06.696 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:06.697 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:06.697 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:06.768 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install cephadm 2024-02-10T06:32:07.172 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:59 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:07.330 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 80 k 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:07.331 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:07.332 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:07.332 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:07.332 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 80 k 2024-02-10T06:32:07.332 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 364 k 2024-02-10T06:32:07.332 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:07.348 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:58 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:07.505 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 80 k 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:07.506 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:07.507 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:07.507 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 80 k 2024-02-10T06:32:07.507 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 364 k 2024-02-10T06:32:07.507 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:07.575 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-17.2.7-505.g72ce7c1d.el9.noarch.rpm 331 kB/s | 80 kB 00:00 2024-02-10T06:32:07.576 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:07.576 INFO:teuthology.orchestra.run.smithi181.stdout:Total 328 kB/s | 80 kB 00:00 2024-02-10T06:32:07.577 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:07.581 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:07.581 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:07.583 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:07.583 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:07.764 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm-17.2.7-505.g72ce7c1d.el9.noarch.rpm 313 kB/s | 80 kB 00:00 2024-02-10T06:32:07.765 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:07.765 INFO:teuthology.orchestra.run.smithi073.stdout:Total 311 kB/s | 80 kB 00:00 2024-02-10T06:32:07.765 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:07.770 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:07.770 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:07.772 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:07.772 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:07.848 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.221 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.256 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.353 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.738 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:08.911 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:08.912 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:08.912 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:08.912 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:08.912 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:09.066 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:32:09.066 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:09.066 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:09.066 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:09.066 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:09.066 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:09.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T06:32:09.280 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T06:32:09.701 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:01 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:09.857 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 137 k 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:09.858 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:09.859 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:09.859 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:09.859 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:09.859 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 137 k 2024-02-10T06:32:09.859 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 379 k 2024-02-10T06:32:09.859 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:09.880 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:00 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:10.037 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 137 k 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 137 k 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 379 k 2024-02-10T06:32:10.038 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:10.289 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-17.2.7-505.g72ce7c1 318 kB/s | 137 kB 00:00 2024-02-10T06:32:10.290 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:10.290 INFO:teuthology.orchestra.run.smithi181.stdout:Total 317 kB/s | 137 kB 00:00 2024-02-10T06:32:10.290 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:10.297 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:10.297 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:10.348 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:10.349 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:10.453 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-immutable-object-cache-17.2.7-505.g72ce7c1 330 kB/s | 137 kB 00:00 2024-02-10T06:32:10.454 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:10.454 INFO:teuthology.orchestra.run.smithi073.stdout:Total 329 kB/s | 137 kB 00:00 2024-02-10T06:32:10.454 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:10.462 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:10.462 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:10.490 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:10.515 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:10.515 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:10.557 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:32:10.581 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:32:10.582 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:10.582 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T06:32:10.582 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:10.662 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:10.712 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:32:10.737 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:32:10.737 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:10.738 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T06:32:10.738 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:11.599 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:32:11.599 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:11.599 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:11.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:11.599 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:11.599 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:11.724 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:32:11.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:11.725 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:11.725 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:11.725 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:11.725 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:11.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-02-10T06:32:11.968 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr 2024-02-10T06:32:12.447 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:04 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:12.554 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:12.554 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:12.605 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:12.606 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:12.606 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:12.662 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:12.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T06:32:12.714 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:12.715 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:12.715 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:12.783 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T06:32:13.249 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:05 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:13.360 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:04 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:13.405 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 1.7 M 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 23 k 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 15 k 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T06:32:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout:Install 7 Packages 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:13.407 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-02-10T06:32:13.408 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 8.1 M 2024-02-10T06:32:13.408 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:13.517 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:13.518 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:13.518 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:32:13.518 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 1.7 M 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 23 k 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 15 k 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout:Installing weak dependencies: 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:13.519 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:13.520 INFO:teuthology.orchestra.run.smithi073.stdout:Install 7 Packages 2024-02-10T06:32:13.520 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:13.520 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.0 M 2024-02-10T06:32:13.520 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 8.1 M 2024-02-10T06:32:13.520 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:13.760 INFO:teuthology.orchestra.run.smithi181.stdout:(1/7): ceph-prometheus-alerts-17.2.7-505.g72ce7 69 kB/s | 15 kB 00:00 2024-02-10T06:32:13.786 INFO:teuthology.orchestra.run.smithi181.stdout:(2/7): ceph-grafana-dashboards-17.2.7-505.g72ce 99 kB/s | 23 kB 00:00 2024-02-10T06:32:13.867 INFO:teuthology.orchestra.run.smithi073.stdout:(1/7): ceph-grafana-dashboards-17.2.7-505.g72ce 118 kB/s | 23 kB 00:00 2024-02-10T06:32:13.892 INFO:teuthology.orchestra.run.smithi073.stdout:(2/7): ceph-prometheus-alerts-17.2.7-505.g72ce7 65 kB/s | 15 kB 00:00 2024-02-10T06:32:13.911 INFO:teuthology.orchestra.run.smithi181.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 60 kB/s | 9.0 kB 00:00 2024-02-10T06:32:13.970 INFO:teuthology.orchestra.run.smithi181.stdout:(4/7): ceph-mgr-dashboard-17.2.7-505.g72ce7c1d. 4.0 MB/s | 1.7 MB 00:00 2024-02-10T06:32:14.012 INFO:teuthology.orchestra.run.smithi181.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 181 kB/s | 41 kB 00:00 2024-02-10T06:32:14.028 INFO:teuthology.orchestra.run.smithi181.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 263 kB/s | 31 kB 00:00 2024-02-10T06:32:14.093 INFO:teuthology.orchestra.run.smithi073.stdout:(3/7): ceph-mgr-dashboard-17.2.7-505.g72ce7c1d. 4.0 MB/s | 1.7 MB 00:00 2024-02-10T06:32:14.118 INFO:teuthology.orchestra.run.smithi073.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 36 kB/s | 9.0 kB 00:00 2024-02-10T06:32:14.237 INFO:teuthology.orchestra.run.smithi181.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 706 kB/s | 188 kB 00:00 2024-02-10T06:32:14.239 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:14.240 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-02-10T06:32:14.240 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:14.247 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:14.247 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:14.311 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:14.311 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:14.402 INFO:teuthology.orchestra.run.smithi073.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 80 kB/s | 41 kB 00:00 2024-02-10T06:32:14.585 INFO:teuthology.orchestra.run.smithi073.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 62 kB/s | 31 kB 00:00 2024-02-10T06:32:14.623 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-10T06:32:14.705 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-10T06:32:14.751 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-10T06:32:14.792 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-10T06:32:14.821 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 5/7 2024-02-10T06:32:14.860 INFO:teuthology.orchestra.run.smithi073.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 254 kB/s | 188 kB 00:00 2024-02-10T06:32:14.864 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:14.864 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-02-10T06:32:14.865 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:14.874 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:14.874 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:14.946 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:14.947 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 6/7 2024-02-10T06:32:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:32:15.254 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:15.328 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-10T06:32:15.362 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-10T06:32:15.416 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-10T06:32:15.449 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-10T06:32:15.477 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 5/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 1/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 3/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-10T06:32:15.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-10T06:32:15.706 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 6/7 2024-02-10T06:32:15.730 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:32:15.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-10T06:32:15.950 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:15.950 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:15.950 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:15.950 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:15.950 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:15.951 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-10T06:32:15.951 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-10T06:32:15.951 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T06:32:15.951 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T06:32:15.951 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:15.951 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:16.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T06:32:16.224 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:32:16.224 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 1/7 2024-02-10T06:32:16.225 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2/7 2024-02-10T06:32:16.225 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 3/7 2024-02-10T06:32:16.225 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-10T06:32:16.225 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-10T06:32:16.225 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-10T06:32:16.534 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T06:32:16.535 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T06:32:16.536 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:16.536 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:16.770 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:08 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:16.782 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T06:32:16.930 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================== 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================== 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 7.4 M 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T06:32:16.931 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================== 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-02-10T06:32:16.932 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:16.933 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 37 M 2024-02-10T06:32:16.933 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 209 M 2024-02-10T06:32:16.933 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:17.128 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 246 kB/s | 32 kB 00:00 2024-02-10T06:32:17.153 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 735 kB/s | 17 kB 00:00 2024-02-10T06:32:17.178 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-02-10T06:32:17.229 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-02-10T06:32:17.360 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:08 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:17.404 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-02-10T06:32:17.471 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-10T06:32:17.496 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-10T06:32:17.522 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout:=========================================================================================== 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout:=========================================================================================== 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 7.4 M 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T06:32:17.523 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout:=========================================================================================== 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout:Install 12 Packages 2024-02-10T06:32:17.524 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:17.525 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 37 M 2024-02-10T06:32:17.525 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 209 M 2024-02-10T06:32:17.525 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:17.646 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.3 MB/s | 802 kB 00:00 2024-02-10T06:32:17.805 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-10T06:32:17.825 INFO:teuthology.orchestra.run.smithi073.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-02-10T06:32:17.830 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-10T06:32:17.851 INFO:teuthology.orchestra.run.smithi073.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 727 kB/s | 17 kB 00:00 2024-02-10T06:32:17.875 INFO:teuthology.orchestra.run.smithi073.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-02-10T06:32:17.951 INFO:teuthology.orchestra.run.smithi073.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-02-10T06:32:18.118 INFO:teuthology.orchestra.run.smithi073.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-02-10T06:32:18.193 INFO:teuthology.orchestra.run.smithi073.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-10T06:32:18.218 INFO:teuthology.orchestra.run.smithi073.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-02-10T06:32:18.289 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-5 5.7 MB/s | 7.4 MB 00:01 2024-02-10T06:32:18.335 INFO:teuthology.orchestra.run.smithi073.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 802 kB 00:00 2024-02-10T06:32:18.528 INFO:teuthology.orchestra.run.smithi073.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-10T06:32:18.561 INFO:teuthology.orchestra.run.smithi073.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 883 kB/s | 194 kB 00:00 2024-02-10T06:32:18.686 INFO:teuthology.orchestra.run.smithi073.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-02-10T06:32:19.005 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 204 kB/s | 245 kB 00:01 2024-02-10T06:32:19.009 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:19.009 INFO:teuthology.orchestra.run.smithi181.stdout:Total 18 MB/s | 37 MB 00:02 2024-02-10T06:32:19.030 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:19.057 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:19.057 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:19.095 INFO:teuthology.orchestra.run.smithi073.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 5.3 MB/s | 7.4 MB 00:01 2024-02-10T06:32:19.097 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:19.097 INFO:teuthology.orchestra.run.smithi073.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-10T06:32:19.118 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:19.143 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:19.143 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:19.488 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:19.488 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:19.575 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:19.576 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:20.190 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:20.270 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T06:32:20.273 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:20.302 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T06:32:20.354 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T06:32:20.382 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T06:32:20.395 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:32:20.438 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T06:32:20.477 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:32:20.519 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T06:32:20.615 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:32:20.653 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:32:20.687 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:32:20.718 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:32:20.730 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T06:32:20.787 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T06:32:21.388 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T06:32:21.456 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T06:32:21.507 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T06:32:21.563 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T06:32:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T06:32:22.812 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T06:32:23.787 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T06:32:23.820 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:32:23.831 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T06:32:23.859 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:32:26.343 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:32:26.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 1/12 2024-02-10T06:32:26.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T06:32:26.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:32:26.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T06:32:26.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:32:26.344 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:32:26.344 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T06:32:26.344 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T06:32:26.344 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T06:32:26.344 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T06:32:26.344 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T06:32:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 1/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T06:32:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T06:32:26.771 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:26.772 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T06:32:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:26.934 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T06:32:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T06:32:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T06:32:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T06:32:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T06:32:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:26.936 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:27.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-02-10T06:32:27.306 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-rook 2024-02-10T06:32:27.720 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:19 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:27.879 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:27.880 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 47 k 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T06:32:27.881 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-02-10T06:32:27.882 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:27.883 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-02-10T06:32:27.883 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-02-10T06:32:27.883 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:27.900 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:18 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:28.054 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 189 kB/s | 24 kB 00:00 2024-02-10T06:32:28.059 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 47 k 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:32:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T06:32:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-02-10T06:32:28.062 INFO:teuthology.orchestra.run.smithi073.stdout:Install 13 Packages 2024-02-10T06:32:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:28.063 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.0 M 2024-02-10T06:32:28.064 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 26 M 2024-02-10T06:32:28.064 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:28.071 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-02-10T06:32:28.122 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-02-10T06:32:28.147 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-10T06:32:28.172 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T06:32:28.197 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-02-10T06:32:28.223 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): ceph-mgr-rook-17.2.7-505.g72ce7c1d.el9. 160 kB/s | 47 kB 00:00 2024-02-10T06:32:28.235 INFO:teuthology.orchestra.run.smithi073.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 188 kB/s | 24 kB 00:00 2024-02-10T06:32:28.260 INFO:teuthology.orchestra.run.smithi073.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-02-10T06:32:28.311 INFO:teuthology.orchestra.run.smithi073.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-10T06:32:28.336 INFO:teuthology.orchestra.run.smithi073.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-10T06:32:28.361 INFO:teuthology.orchestra.run.smithi073.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T06:32:28.387 INFO:teuthology.orchestra.run.smithi073.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-02-10T06:32:28.412 INFO:teuthology.orchestra.run.smithi073.stdout:(7/13): ceph-mgr-rook-17.2.7-505.g72ce7c1d.el9. 155 kB/s | 47 kB 00:00 2024-02-10T06:32:28.621 INFO:teuthology.orchestra.run.smithi073.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-02-10T06:32:28.646 INFO:teuthology.orchestra.run.smithi073.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-02-10T06:32:28.647 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 68 kB/s | 32 kB 00:00 2024-02-10T06:32:28.673 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2024-02-10T06:32:28.857 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-02-10T06:32:28.863 INFO:teuthology.orchestra.run.smithi073.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 450 kB/s | 203 kB 00:00 2024-02-10T06:32:28.888 INFO:teuthology.orchestra.run.smithi073.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2024-02-10T06:32:28.974 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 270 kB/s | 203 kB 00:00 2024-02-10T06:32:29.005 INFO:teuthology.orchestra.run.smithi073.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-02-10T06:32:29.021 INFO:teuthology.orchestra.run.smithi073.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 565 kB/s | 90 kB 00:00 2024-02-10T06:32:29.024 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:29.024 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-02-10T06:32:29.025 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:29.039 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:29.039 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:29.040 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-02-10T06:32:29.143 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:29.143 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:29.663 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:29.834 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T06:32:29.890 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T06:32:29.944 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T06:32:29.974 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T06:32:30.015 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T06:32:30.108 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T06:32:30.115 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 712 kB/s | 1.0 MB 00:01 2024-02-10T06:32:30.120 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:30.121 INFO:teuthology.orchestra.run.smithi181.stdout:Total 935 kB/s | 2.0 MB 00:02 2024-02-10T06:32:30.121 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:30.136 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:30.136 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:30.217 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T06:32:30.242 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:30.243 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:30.283 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T06:32:30.772 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:30.944 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T06:32:30.998 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T06:32:30.999 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T06:32:31.055 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T06:32:31.055 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T06:32:31.089 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T06:32:31.093 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T06:32:31.134 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T06:32:31.134 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T06:32:31.160 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:32:31.227 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T06:32:31.336 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T06:32:31.385 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 1/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T06:32:31.958 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T06:32:31.959 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T06:32:31.959 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T06:32:31.959 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T06:32:31.959 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T06:32:31.959 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T06:32:31.959 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T06:32:32.105 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T06:32:32.157 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T06:32:32.183 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T06:32:32.228 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T06:32:32.253 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T06:32:32.342 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:32.343 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:32.588 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T06:32:33.116 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:32:33.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 1/13 2024-02-10T06:32:33.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T06:32:33.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T06:32:33.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T06:32:33.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T06:32:33.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T06:32:33.196 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:24 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:33.369 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 121 k 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T06:32:33.370 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2024-02-10T06:32:33.371 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:33.372 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 6.8 M 2024-02-10T06:32:33.372 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 33 M 2024-02-10T06:32:33.372 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:33.488 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T06:32:33.488 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:33.488 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:33.488 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:33.488 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T06:32:33.488 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:33.489 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:33.723 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-02-10T06:32:33.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T06:32:33.865 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-02-10T06:32:33.915 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): ceph-mgr-cephadm-17.2.7-505.g72ce7c1d.el 330 kB/s | 121 kB 00:00 2024-02-10T06:32:34.182 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-02-10T06:32:34.216 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 170 kB/s | 51 kB 00:00 2024-02-10T06:32:34.316 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:26 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:34.324 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 913 kB/s | 548 kB 00:00 2024-02-10T06:32:34.326 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:34.327 INFO:teuthology.orchestra.run.smithi073.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-02-10T06:32:34.327 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:34.335 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:34.336 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:34.425 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:34.426 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:34.475 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:34.476 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 121 k 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T06:32:34.477 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 6.8 M 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-02-10T06:32:34.478 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:34.729 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:34.775 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-10T06:32:34.919 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T06:32:34.926 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-10T06:32:34.976 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T06:32:34.992 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): ceph-mgr-cephadm-17.2.7-505.g72ce7c1d.el 322 kB/s | 121 kB 00:00 2024-02-10T06:32:35.018 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-02-10T06:32:35.043 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 493 kB/s | 58 kB 00:00 2024-02-10T06:32:35.093 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 508 kB/s | 51 kB 00:00 2024-02-10T06:32:35.095 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:35.095 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-02-10T06:32:35.096 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:35.104 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:35.104 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:35.194 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:35.194 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:35.497 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T06:32:35.499 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:35.587 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T06:32:35.650 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T06:32:35.657 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T06:32:35.680 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:32:35.703 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T06:32:36.212 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T06:32:36.289 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T06:32:36.343 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T06:32:36.366 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:32:36.554 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:32:36.554 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/6 2024-02-10T06:32:36.555 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T06:32:36.555 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T06:32:36.555 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T06:32:36.555 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T06:32:36.873 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T06:32:36.873 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:36.873 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:36.873 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:36.874 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:37.140 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-fuse 2024-02-10T06:32:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:32:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/6 2024-02-10T06:32:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T06:32:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T06:32:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T06:32:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T06:32:37.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T06:32:37.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T06:32:37.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T06:32:37.471 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:37.471 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:37.725 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:28 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:37.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-02-10T06:32:37.883 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 812 k 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T06:32:37.884 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:37.885 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:37.885 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:37.885 INFO:teuthology.orchestra.run.smithi073.stdout:Install 2 Packages 2024-02-10T06:32:37.885 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:37.885 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 891 k 2024-02-10T06:32:37.885 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 2.6 M 2024-02-10T06:32:37.886 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:38.010 INFO:teuthology.orchestra.run.smithi073.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 635 kB/s | 79 kB 00:00 2024-02-10T06:32:38.243 INFO:teuthology.orchestra.run.smithi073.stdout:(2/2): ceph-fuse-17.2.7-505.g72ce7c1d.el9.x86_6 2.2 MB/s | 812 kB 00:00 2024-02-10T06:32:38.244 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:38.244 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.4 MB/s | 891 kB 00:00 2024-02-10T06:32:38.245 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:38.264 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:38.264 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:38.339 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:38.339 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:38.357 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:38.515 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 812 k 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:38.516 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:38.517 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:38.517 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-02-10T06:32:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:38.517 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 891 k 2024-02-10T06:32:38.517 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.6 M 2024-02-10T06:32:38.517 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:38.526 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:38.636 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 662 kB/s | 79 kB 00:00 2024-02-10T06:32:38.654 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T06:32:38.734 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:32:38.878 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-17.2.7-505.g72ce7c1d.el9.x86_6 2.2 MB/s | 812 kB 00:00 2024-02-10T06:32:38.879 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:38.880 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.4 MB/s | 891 kB 00:00 2024-02-10T06:32:38.880 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:38.900 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:38.900 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:38.975 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:38.975 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:39.163 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T06:32:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:32:39.376 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:32:39.376 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/2 2024-02-10T06:32:39.740 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T06:32:39.740 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:39.740 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:39.740 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T06:32:39.740 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:39.740 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:39.992 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados-devel 2024-02-10T06:32:40.050 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:32:40.050 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/2 2024-02-10T06:32:40.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T06:32:40.343 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:40.344 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:40.344 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T06:32:40.344 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:40.344 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:40.588 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:31 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:40.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-02-10T06:32:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 126 k 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 126 k 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 448 k 2024-02-10T06:32:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:40.981 INFO:teuthology.orchestra.run.smithi073.stdout:librados-devel-17.2.7-505.g72ce7c1d.el9.x86_64. 546 kB/s | 126 kB 00:00 2024-02-10T06:32:40.981 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:40.981 INFO:teuthology.orchestra.run.smithi073.stdout:Total 543 kB/s | 126 kB 00:00 2024-02-10T06:32:40.981 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:40.987 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:40.987 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:41.043 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:41.043 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:41.182 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:41.214 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:41.279 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:41.371 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:41.372 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:41.372 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-10T06:32:41.372 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:41.372 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 126 k 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 126 k 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 448 k 2024-02-10T06:32:41.373 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:41.624 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-17.2.7-505.g72ce7c1d.el9.x86_64. 504 kB/s | 126 kB 00:00 2024-02-10T06:32:41.624 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:41.625 INFO:teuthology.orchestra.run.smithi181.stdout:Total 501 kB/s | 126 kB 00:00 2024-02-10T06:32:41.625 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:41.630 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:41.630 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:41.684 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:41.685 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:41.746 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:41.825 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:42.088 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:42.089 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:42.089 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:42.089 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:42.089 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:42.089 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:42.365 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs2 2024-02-10T06:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:42.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:42.679 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:42.679 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:42.679 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:42.680 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:42.680 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:42.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-02-10T06:32:42.964 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:43.072 INFO:teuthology.orchestra.run.smithi073.stdout:Package libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:43.125 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:43.126 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:43.127 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:43.190 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs-devel 2024-02-10T06:32:43.539 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:43.650 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:43.705 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:43.706 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:43.706 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:43.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-02-10T06:32:43.792 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:34 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:43.950 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:43.951 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:43.951 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-02-10T06:32:43.951 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:43.951 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 25 k 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 110 k 2024-02-10T06:32:43.952 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:44.142 INFO:teuthology.orchestra.run.smithi073.stdout:libcephfs-devel-17.2.7-505.g72ce7c1d.el9.x86_64 131 kB/s | 25 kB 00:00 2024-02-10T06:32:44.143 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:44.143 INFO:teuthology.orchestra.run.smithi073.stdout:Total 130 kB/s | 25 kB 00:00 2024-02-10T06:32:44.143 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:44.146 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:44.147 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:44.148 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:44.149 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:44.235 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:44.340 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:44.363 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:44.521 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:44.522 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 25 k 2024-02-10T06:32:44.523 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 110 k 2024-02-10T06:32:44.523 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:44.712 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-17.2.7-505.g72ce7c1d.el9.x86_64 133 kB/s | 25 kB 00:00 2024-02-10T06:32:44.712 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:44.712 INFO:teuthology.orchestra.run.smithi181.stdout:Total 132 kB/s | 25 kB 00:00 2024-02-10T06:32:44.712 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:44.715 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:44.715 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:44.716 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:44.717 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:44.747 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:44.804 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:44.908 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:45.027 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:45.027 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:45.027 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:45.027 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:45.027 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:45.027 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:45.239 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados2 2024-02-10T06:32:45.291 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:45.572 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:45.572 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:45.572 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:45.572 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:45.572 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:45.572 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:45.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-02-10T06:32:45.823 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:45.930 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:45.982 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:45.983 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:45.983 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:46.048 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librbd1 2024-02-10T06:32:46.329 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:46.435 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:46.487 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:46.488 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:46.488 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:46.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-02-10T06:32:46.629 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:46.735 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:46.789 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:46.790 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:46.790 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:46.865 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rados 2024-02-10T06:32:47.142 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:47.248 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:47.301 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:47.302 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:47.302 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:47.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-02-10T06:32:47.446 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:38 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:47.553 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:47.606 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:47.607 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:47.607 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:47.673 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rgw 2024-02-10T06:32:47.959 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:48.065 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:48.118 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:48.119 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:48.119 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:48.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-02-10T06:32:48.258 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:48.365 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:48.418 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:48.419 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:48.419 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:48.493 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-cephfs 2024-02-10T06:32:48.770 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:48.879 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:48.933 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:48.933 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:48.934 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:48.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-02-10T06:32:49.081 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:49.197 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:49.250 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:49.251 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:49.251 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:49.316 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rbd 2024-02-10T06:32:49.586 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:41 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:49.692 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:49.747 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:49.747 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:49.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-02-10T06:32:49.897 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:50.004 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:50.058 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:50.059 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-02-10T06:32:50.059 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:50.125 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-fuse 2024-02-10T06:32:50.395 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:42 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:50.503 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:32:50.557 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:50.558 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-10T06:32:50.558 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:50.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-02-10T06:32:50.709 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:41 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:50.871 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 81 k 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:50.872 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:50.873 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:50.873 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:50.873 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 81 k 2024-02-10T06:32:50.873 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 222 k 2024-02-10T06:32:50.873 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:51.120 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-fuse-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 325 kB/s | 81 kB 00:00 2024-02-10T06:32:51.121 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:51.121 INFO:teuthology.orchestra.run.smithi073.stdout:Total 324 kB/s | 81 kB 00:00 2024-02-10T06:32:51.121 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:51.128 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:51.129 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:51.182 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:51.182 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:51.213 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:51.315 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:51.372 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 81 k 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:51.373 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:51.374 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:51.374 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:51.374 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 81 k 2024-02-10T06:32:51.374 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 222 k 2024-02-10T06:32:51.374 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:51.435 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:51.613 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 339 kB/s | 81 kB 00:00 2024-02-10T06:32:51.614 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:51.614 INFO:teuthology.orchestra.run.smithi181.stdout:Total 337 kB/s | 81 kB 00:00 2024-02-10T06:32:51.614 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:51.621 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:51.621 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:51.675 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:51.676 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:51.833 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:51.920 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:52.186 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:52.186 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:52.186 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:52.186 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:52.186 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:52.186 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:52.312 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:52.421 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-mirror 2024-02-10T06:32:52.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:52.584 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:52.585 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:52.585 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:52.585 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:52.585 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:52.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-02-10T06:32:53.011 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:53.177 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:53.178 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:53.178 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-02-10T06:32:53.178 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:53.178 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:53.178 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.9 M 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 12 M 2024-02-10T06:32:53.179 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:53.410 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:53.574 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:53.575 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:53.575 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-10T06:32:53.575 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:53.575 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.9 M 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 12 M 2024-02-10T06:32:53.576 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:53.732 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-mirror-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 5.3 MB/s | 2.9 MB 00:00 2024-02-10T06:32:53.733 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:53.733 INFO:teuthology.orchestra.run.smithi073.stdout:Total 5.3 MB/s | 2.9 MB 00:00 2024-02-10T06:32:53.733 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:53.742 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:53.742 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:53.820 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:53.821 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:54.133 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 5.3 MB/s | 2.9 MB 00:00 2024-02-10T06:32:54.133 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:54.134 INFO:teuthology.orchestra.run.smithi181.stdout:Total 5.3 MB/s | 2.9 MB 00:00 2024-02-10T06:32:54.134 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:54.143 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:54.143 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:54.219 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:54.219 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:54.237 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:54.308 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:54.331 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:54.332 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:54.332 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T06:32:54.332 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:32:54.332 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:32:54.332 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:54.632 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:54.686 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:54.709 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:54.709 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:32:54.709 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T06:32:54.710 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:32:54.710 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:32:54.710 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:55.418 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:55.418 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:55.418 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:55.418 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:55.418 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:55.418 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:55.659 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-nbd 2024-02-10T06:32:55.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:55.954 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:55.954 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:55.954 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:55.954 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:55.954 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:56.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-02-10T06:32:56.256 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 06:31:09 AM UTC. 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 164 k 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-02-10T06:32:56.423 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-02-10T06:32:56.424 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:56.424 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 164 k 2024-02-10T06:32:56.424 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 477 k 2024-02-10T06:32:56.424 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-02-10T06:32:56.669 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-nbd-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 671 kB/s | 164 kB 00:00 2024-02-10T06:32:56.670 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:56.670 INFO:teuthology.orchestra.run.smithi073.stdout:Total 667 kB/s | 164 kB 00:00 2024-02-10T06:32:56.670 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-02-10T06:32:56.678 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-02-10T06:32:56.678 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-02-10T06:32:56.734 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-02-10T06:32:56.734 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-02-10T06:32:56.853 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:48 ago on Sat 10 Feb 2024 06:31:08 AM UTC. 2024-02-10T06:32:56.876 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-02-10T06:32:56.983 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:57.012 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-10T06:32:57.012 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 164 k 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 164 k 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 477 k 2024-02-10T06:32:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-10T06:32:57.439 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 387 kB/s | 164 kB 00:00 2024-02-10T06:32:57.440 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:32:57.440 INFO:teuthology.orchestra.run.smithi181.stdout:Total 385 kB/s | 164 kB 00:00 2024-02-10T06:32:57.441 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-10T06:32:57.448 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-10T06:32:57.448 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-10T06:32:57.480 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:57.503 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-10T06:32:57.503 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-10T06:32:57.651 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-10T06:32:57.761 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:57.828 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:57.828 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:57.828 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-02-10T06:32:57.828 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:57.828 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:32:57.828 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-02-10T06:32:58.084 DEBUG:teuthology.parallel:result is None 2024-02-10T06:32:58.210 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:58.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:32:58.543 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:58.543 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-10T06:32:58.543 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:32:58.543 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:32:58.543 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-10T06:32:58.776 DEBUG:teuthology.parallel:result is None 2024-02-10T06:32:58.776 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:32:58.777 INFO:teuthology.packaging:ref: None 2024-02-10T06:32:58.777 INFO:teuthology.packaging:tag: None 2024-02-10T06:32:58.777 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:32:58.777 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:32:59.056 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T06:32:59.080 INFO:teuthology.orchestra.run.smithi073.stdout:17.2.7-505.g72ce7c1d.el9 2024-02-10T06:32:59.080 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-505.g72ce7c1d.el9 2024-02-10T06:32:59.080 INFO:teuthology.task.install:The correct ceph version 17.2.7-505.g72ce7c1d is installed. 2024-02-10T06:32:59.082 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:32:59.082 INFO:teuthology.packaging:ref: None 2024-02-10T06:32:59.082 INFO:teuthology.packaging:tag: None 2024-02-10T06:32:59.083 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:32:59.083 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:59.083 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:32:59.304 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T06:32:59.326 INFO:teuthology.orchestra.run.smithi181.stdout:17.2.7-505.g72ce7c1d.el9 2024-02-10T06:32:59.326 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-505.g72ce7c1d.el9 2024-02-10T06:32:59.327 INFO:teuthology.task.install:The correct ceph version 17.2.7-505.g72ce7c1d is installed. 2024-02-10T06:32:59.328 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T06:32:59.329 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:32:59.329 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T06:32:59.362 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:32:59.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T06:32:59.395 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T06:32:59.396 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:32:59.396 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T06:32:59.432 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T06:32:59.502 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:32:59.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T06:32:59.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T06:32:59.597 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T06:32:59.598 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:32:59.598 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T06:32:59.627 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T06:32:59.696 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:32:59.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T06:32:59.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T06:32:59.790 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T06:32:59.791 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:32:59.791 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T06:32:59.819 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T06:32:59.887 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:32:59.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T06:32:59.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T06:32:59.980 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:32:59.994 INFO:teuthology.task.print:**** done install task... 2024-02-10T06:32:59.994 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T06:33:00.121 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T06:33:00.121 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-10T06:33:00.121 INFO:tasks.cephadm:Cluster fsid is 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:00.121 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T06:33:00.122 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-10T06:33:00.122 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi073': '172.21.15.73', 'mon.smithi181': '172.21.15.181'} 2024-02-10T06:33:00.122 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T06:33:00.122 DEBUG:teuthology.orchestra.run.smithi073:> sudo hostname $(hostname -s) 2024-02-10T06:33:00.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-02-10T06:33:00.177 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-10T06:33:00.177 DEBUG:teuthology.orchestra.run.smithi073:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T06:33:00.415 INFO:teuthology.orchestra.run.smithi073.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 10 06:33 /home/ubuntu/cephtest/cephadm 2024-02-10T06:33:00.415 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T06:33:00.587 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 10 06:33 /home/ubuntu/cephtest/cephadm 2024-02-10T06:33:00.588 DEBUG:teuthology.orchestra.run.smithi073:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T06:33:00.603 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T06:33:00.628 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-10T06:33:00.628 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-10T06:33:00.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-10T06:33:01.500 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:33:01.563 INFO:teuthology.orchestra.run.smithi073.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:33:26.864 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-02-10T06:33:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_version": "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)", 2024-02-10T06:33:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "image_id": "a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5", 2024-02-10T06:33:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "repo_digests": [ 2024-02-10T06:33:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc" 2024-02-10T06:33:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-02-10T06:33:26.865 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-02-10T06:33:29.657 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-10T06:33:29.657 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)", 2024-02-10T06:33:29.657 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5", 2024-02-10T06:33:29.657 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-02-10T06:33:29.657 INFO:teuthology.orchestra.run.smithi181.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc" 2024-02-10T06:33:29.657 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-10T06:33:29.658 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-10T06:33:29.677 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph 2024-02-10T06:33:29.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-02-10T06:33:29.741 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /etc/ceph 2024-02-10T06:33:29.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-02-10T06:33:29.808 INFO:tasks.cephadm:Writing seed config... 2024-02-10T06:33:29.809 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-10T06:33:29.810 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-10T06:33:29.811 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T06:33:29.812 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-10T06:33:29.813 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:33:29.813 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T06:33:29.837 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3c2b87a2-c7de-11ee-95b8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-10T06:33:29.838 DEBUG:teuthology.orchestra.run.smithi073:mon.smithi073> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service 2024-02-10T06:33:29.880 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T06:33:29.880 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.73 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:33:30.090 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:33:30.090 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '3c2b87a2-c7de-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.73', '--skip-admin-label'] 2024-02-10T06:33:30.114 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T06:33:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-10T06:33:30.115 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying podman|docker is present... 2024-02-10T06:33:30.137 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T06:33:30.138 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying lvm2 is present... 2024-02-10T06:33:30.138 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying time synchronization is in place... 2024-02-10T06:33:30.145 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T06:33:30.145 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T06:33:30.151 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T06:33:30.151 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-02-10T06:33:30.157 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout enabled 2024-02-10T06:33:30.163 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout active 2024-02-10T06:33:30.163 INFO:teuthology.orchestra.run.smithi073.stdout:Unit chronyd.service is enabled and running 2024-02-10T06:33:30.163 INFO:teuthology.orchestra.run.smithi073.stdout:Repeating the final host check... 2024-02-10T06:33:30.187 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T06:33:30.187 INFO:teuthology.orchestra.run.smithi073.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-10T06:33:30.187 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl is present 2024-02-10T06:33:30.187 INFO:teuthology.orchestra.run.smithi073.stdout:lvcreate is present 2024-02-10T06:33:30.194 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T06:33:30.194 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T06:33:30.199 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T06:33:30.199 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-02-10T06:33:30.206 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout enabled 2024-02-10T06:33:30.212 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout active 2024-02-10T06:33:30.212 INFO:teuthology.orchestra.run.smithi073.stdout:Unit chronyd.service is enabled and running 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Host looks OK 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Cluster fsid: 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Acquiring lock 140675966183072 on /run/cephadm/3c2b87a2-c7de-11ee-95b8-87774f69a715.lock 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Lock 140675966183072 acquired on /run/cephadm/3c2b87a2-c7de-11ee-95b8-87774f69a715.lock 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying IP 172.21.15.73 port 3300 ... 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying IP 172.21.15.73 port 6789 ... 2024-02-10T06:33:30.213 INFO:teuthology.orchestra.run.smithi073.stdout:Base mon IP(s) is [172.21.15.73:3300, 172.21.15.73:6789], mon addrv is [v2:172.21.15.73:3300,v1:172.21.15.73:6789] 2024-02-10T06:33:30.216 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.73 metric 100 2024-02-10T06:33:30.216 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.73 metric 100 2024-02-10T06:33:30.219 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T06:33:30.219 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T06:33:30.219 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1736sec hoplimit 64 pref medium 2024-02-10T06:33:30.221 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T06:33:30.221 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce7d/64 scope link 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:Mon IP `172.21.15.73` is in CIDR network `172.21.0.0/20` 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:Mon IP `172.21.15.73` is in CIDR network `172.21.0.0/20` 2024-02-10T06:33:30.222 INFO:teuthology.orchestra.run.smithi073.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-10T06:33:30.223 INFO:teuthology.orchestra.run.smithi073.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T06:33:30.224 INFO:teuthology.orchestra.run.smithi073.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:33:30.690 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 2024-02-10T06:33:30.691 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:33:30.691 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Getting image source signatures 2024-02-10T06:33:30.691 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Copying blob sha256:a61c475ff6e77d421bceb2aff73188667ff70f4da71ee8d2a5b09b5c5d4ca045 2024-02-10T06:33:30.691 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T06:33:30.691 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Copying config sha256:a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 2024-02-10T06:33:30.691 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-10T06:33:31.570 INFO:teuthology.orchestra.run.smithi073.stdout:ceph: stdout ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable) 2024-02-10T06:33:31.571 INFO:teuthology.orchestra.run.smithi073.stdout:Ceph version: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable) 2024-02-10T06:33:31.571 INFO:teuthology.orchestra.run.smithi073.stdout:Extracting ceph user uid/gid from container image... 2024-02-10T06:33:32.920 INFO:teuthology.orchestra.run.smithi073.stdout:stat: stdout 167 167 2024-02-10T06:33:32.920 INFO:teuthology.orchestra.run.smithi073.stdout:Creating initial keys... 2024-02-10T06:33:42.181 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph-authtool: stdout AQC/GMdlYYUfLRAAeKHivRQiky/UU8rx25CUMQ== 2024-02-10T06:33:45.431 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph-authtool: stdout AQDIGMdlIAgmNRAA0wZry9fhwSzc4EanoTN1+g== 2024-02-10T06:33:46.295 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph-authtool: stdout AQDJGMdlZpn7KxAA7EHMmMT0IorEH6JWGBX/ww== 2024-02-10T06:33:46.296 INFO:teuthology.orchestra.run.smithi073.stdout:Creating initial monmap... 2024-02-10T06:33:47.168 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool for smithi073 [v2:172.21.15.73:3300,v1:172.21.15.73:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:setting min_mon_release = octopus 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: set fsid to 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:33:47.169 INFO:teuthology.orchestra.run.smithi073.stdout:Creating mon... 2024-02-10T06:33:48.049 INFO:teuthology.orchestra.run.smithi073.stdout:create mon.smithi073 on 2024-02-10T06:33:48.262 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-10T06:33:48.409 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T06:33:48.555 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715.target. 2024-02-10T06:33:48.556 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715.target. 2024-02-10T06:33:48.744 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073 2024-02-10T06:33:48.744 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service: Unit ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service not loaded. 2024-02-10T06:33:48.907 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715.target.wants/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service → /etc/systemd/system/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@.service. 2024-02-10T06:33:49.432 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:49 smithi073 podman[28676]: 2024-02-10 06:33:49.369419178 +0000 UTC m=+0.347876520 container start 6fca2a65a47e3004c2d2356167d701caa65c50ff7012908117e7f9d9365da68f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T06:33:49.432 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:49 smithi073 ceph-mon[28694]: mkfs 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:49.471 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:33:49.472 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-02-10T06:33:49.479 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:33:49.479 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-02-10T06:33:49.479 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-02-10T06:33:49.479 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T06:33:49.479 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mon to start... 2024-02-10T06:33:49.479 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mon... 2024-02-10T06:33:49.743 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:49 smithi073 ceph-mon[28694]: mon.smithi073 is new leader, mons smithi073 in quorum (ranks 0) 2024-02-10T06:33:49.743 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:49 smithi073 bash[28676]: 6fca2a65a47e3004c2d2356167d701caa65c50ff7012908117e7f9d9365da68f 2024-02-10T06:33:49.743 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:49 smithi073 systemd[1]: Started Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:33:50.774 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:50 smithi073 ceph-mon[28694]: mon.smithi073 is new leader, mons smithi073 in quorum (ranks 0) 2024-02-10T06:33:50.774 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:50 smithi073 ceph-mon[28694]: monmap e1: 1 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:33:50.774 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:50 smithi073 ceph-mon[28694]: fsmap 2024-02-10T06:33:50.774 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:50 smithi073 ceph-mon[28694]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T06:33:50.774 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:50 smithi073 ceph-mon[28694]: mgrmap e1: no daemons active 2024-02-10T06:33:50.774 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:50 smithi073 ceph-mon[28694]: from='client.? 172.21.15.73:0/519847378' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:33:50.777 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout cluster: 2024-02-10T06:33:50.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout id: 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout services: 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi073 (age 0.663693s) 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout data: 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout pgs: 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:50.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f719283b700 1 Processor -- start 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f719283b700 1 -- start start 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f719283b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f719283b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c101fc0 con 0x7f718c101670 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f718bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f718bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53940/0 (socket says 172.21.15.73:53940) 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.014+0000 7f718bfff700 1 -- 172.21.15.73:0/536710674 learned_addr learned my addr 172.21.15.73:0/536710674 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f718bfff700 1 -- 172.21.15.73:0/536710674 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c102100 con 0x7f718c101670 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f718bfff700 1 --2- 172.21.15.73:0/536710674 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c101a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f717c0098e0 tx=0x7f717c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa63adf442d6e07f server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f718affd700 1 -- 172.21.15.73:0/536710674 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f717c00b860 con 0x7f718c101670 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f718affd700 1 -- 172.21.15.73:0/536710674 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c00b9c0 con 0x7f718c101670 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f718affd700 1 -- 172.21.15.73:0/536710674 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f717c00bc90 con 0x7f718c101670 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f719283b700 1 -- 172.21.15.73:0/536710674 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 msgr2=0x7f718c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.015+0000 7f719283b700 1 --2- 172.21.15.73:0/536710674 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c101a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f717c0098e0 tx=0x7f717c009250 comp rx=0 tx=0).stop 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 -- 172.21.15.73:0/536710674 shutdown_connections 2024-02-10T06:33:50.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 --2- 172.21.15.73:0/536710674 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c101a80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 -- 172.21.15.73:0/536710674 >> 172.21.15.73:0/536710674 conn(0x7f718c0fcdd0 msgr2=0x7f718c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 -- 172.21.15.73:0/536710674 shutdown_connections 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 -- 172.21.15.73:0/536710674 wait complete. 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 Processor -- start 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.016+0000 7f719283b700 1 -- start start 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f719283b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c110d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f719283b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c1112d0 con 0x7f718c101670 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f718bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c110d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f718bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c110d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53948/0 (socket says 172.21.15.73:53948) 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f718bfff700 1 -- 172.21.15.73:0/519847378 learned_addr learned my addr 172.21.15.73:0/519847378 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f718bfff700 1 -- 172.21.15.73:0/519847378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c0094f0 con 0x7f718c101670 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f718bfff700 1 --2- 172.21.15.73:0/519847378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c110d90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f717c003080 tx=0x7f717c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f717c01c5c0 con 0x7f718c101670 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c01c720 con 0x7f718c101670 2024-02-10T06:33:50.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f717c01ca30 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718c1114d0 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718c1120c0 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f717c01a450 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.017+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f717c02e790 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.018+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718c05f5c0 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.020+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f717c01a8e0 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.090+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f718c04bf20 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.091+0000 7f7188ff9700 1 -- 172.21.15.73:0/519847378 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f717c01aa80 con 0x7f718c101670 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.091+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 msgr2=0x7f718c110d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.091+0000 7f719283b700 1 --2- 172.21.15.73:0/519847378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c110d90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f717c003080 tx=0x7f717c016000 comp rx=0 tx=0).stop 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.091+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 shutdown_connections 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.091+0000 7f719283b700 1 --2- 172.21.15.73:0/519847378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f718c101670 0x7f718c110d90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.091+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 >> 172.21.15.73:0/519847378 conn(0x7f718c0fcdd0 msgr2=0x7f718c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:50.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.092+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 shutdown_connections 2024-02-10T06:33:50.788 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:50.092+0000 7f719283b700 1 -- 172.21.15.73:0/519847378 wait complete. 2024-02-10T06:33:50.788 INFO:teuthology.orchestra.run.smithi073.stdout:mon is available 2024-02-10T06:33:50.788 INFO:teuthology.orchestra.run.smithi073.stdout:Assimilating anything we can from ceph.conf... 2024-02-10T06:33:51.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:51 smithi073 ceph-mon[28694]: from='client.? 172.21.15.73:0/3125333524' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T06:33:51.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:51 smithi073 ceph-mon[28694]: from='client.? 172.21.15.73:0/3125333524' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout fsid = 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.73:3300,v1:172.21.15.73:6789] 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T06:33:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26e06e3700 1 Processor -- start 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26e06e3700 1 -- start start 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26e06e3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26e06e3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81030c0 con 0x7f26d8102770 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26de47f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26de47f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53960/0 (socket says 172.21.15.73:53960) 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26de47f700 1 -- 172.21.15.73:0/149952982 learned_addr learned my addr 172.21.15.73:0/149952982 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.160+0000 7f26de47f700 1 -- 172.21.15.73:0/149952982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8103200 con 0x7f26d8102770 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26de47f700 1 --2- 172.21.15.73:0/149952982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f26c80098e0 tx=0x7f26c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3148c3bc7cbce6 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26dd47d700 1 -- 172.21.15.73:0/149952982 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c800b860 con 0x7f26d8102770 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26dd47d700 1 -- 172.21.15.73:0/149952982 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c800b9c0 con 0x7f26d8102770 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26dd47d700 1 -- 172.21.15.73:0/149952982 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c800bc90 con 0x7f26d8102770 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26e06e3700 1 -- 172.21.15.73:0/149952982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 msgr2=0x7f26d8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26e06e3700 1 --2- 172.21.15.73:0/149952982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f26c80098e0 tx=0x7f26c8009250 comp rx=0 tx=0).stop 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26e06e3700 1 -- 172.21.15.73:0/149952982 shutdown_connections 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26e06e3700 1 --2- 172.21.15.73:0/149952982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8102b80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26e06e3700 1 -- 172.21.15.73:0/149952982 >> 172.21.15.73:0/149952982 conn(0x7f26d80fded0 msgr2=0x7f26d8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.161+0000 7f26e06e3700 1 -- 172.21.15.73:0/149952982 shutdown_connections 2024-02-10T06:33:52.035 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26e06e3700 1 -- 172.21.15.73:0/149952982 wait complete. 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26e06e3700 1 Processor -- start 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26e06e3700 1 -- start start 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26e06e3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8110d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26e06e3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81112d0 con 0x7f26d8102770 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26de47f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8110d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26de47f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8110d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53964/0 (socket says 172.21.15.73:53964) 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.162+0000 7f26de47f700 1 -- 172.21.15.73:0/3125333524 learned_addr learned my addr 172.21.15.73:0/3125333524 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26de47f700 1 -- 172.21.15.73:0/3125333524 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c80094f0 con 0x7f26d8102770 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26de47f700 1 --2- 172.21.15.73:0/3125333524 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8110d90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f26c8003080 tx=0x7f26c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c801c5c0 con 0x7f26d8102770 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c801c720 con 0x7f26d8102770 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d81114d0 con 0x7f26d8102770 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c801ca30 con 0x7f26d8102770 2024-02-10T06:33:52.036 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d81120c0 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f26c801a450 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.163+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f26c802e790 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.164+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d805f5c0 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.166+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f26c801a890 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.232+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f26d8103b90 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.236+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f26c801aa30 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.236+0000 7f26ceffd700 1 -- 172.21.15.73:0/3125333524 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f26c8024e30 con 0x7f26d8102770 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 msgr2=0x7f26d8110d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 --2- 172.21.15.73:0/3125333524 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8110d90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f26c8003080 tx=0x7f26c8016000 comp rx=0 tx=0).stop 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 shutdown_connections 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 --2- 172.21.15.73:0/3125333524 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26d8102770 0x7f26d8110d90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 >> 172.21.15.73:0/3125333524 conn(0x7f26d80fded0 msgr2=0x7f26d81054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 shutdown_connections 2024-02-10T06:33:52.037 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:51.237+0000 7f26e06e3700 1 -- 172.21.15.73:0/3125333524 wait complete. 2024-02-10T06:33:52.038 INFO:teuthology.orchestra.run.smithi073.stdout:Generating new minimal ceph.conf... 2024-02-10T06:33:52.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:52 smithi073 ceph-mon[28694]: from='client.? 172.21.15.73:0/2117854176' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:53.998 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db703b700 1 Processor -- start 2024-02-10T06:33:53.998 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db703b700 1 -- start start 2024-02-10T06:33:53.998 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db703b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:53.998 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db703b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db01058c0 con 0x7f0db0076c20 2024-02-10T06:33:53.998 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db4dd7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db4dd7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53968/0 (socket says 172.21.15.73:53968) 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.459+0000 7f0db4dd7700 1 -- 172.21.15.73:0/3036770761 learned_addr learned my addr 172.21.15.73:0/3036770761 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.460+0000 7f0db4dd7700 1 -- 172.21.15.73:0/3036770761 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0105a00 con 0x7f0db0076c20 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.460+0000 7f0db4dd7700 1 --2- 172.21.15.73:0/3036770761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0105380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0da40098e0 tx=0x7f0da4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bac22e455d569d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.460+0000 7f0daf7fe700 1 -- 172.21.15.73:0/3036770761 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0da400b860 con 0x7f0db0076c20 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.460+0000 7f0daf7fe700 1 -- 172.21.15.73:0/3036770761 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0da400be50 con 0x7f0db0076c20 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.460+0000 7f0daf7fe700 1 -- 172.21.15.73:0/3036770761 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0da4010380 con 0x7f0db0076c20 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 -- 172.21.15.73:0/3036770761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 msgr2=0x7f0db0105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 --2- 172.21.15.73:0/3036770761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0105380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0da40098e0 tx=0x7f0da4009250 comp rx=0 tx=0).stop 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 -- 172.21.15.73:0/3036770761 shutdown_connections 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 --2- 172.21.15.73:0/3036770761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0105380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:53.999 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 -- 172.21.15.73:0/3036770761 >> 172.21.15.73:0/3036770761 conn(0x7f0db00fcdd0 msgr2=0x7f0db00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 -- 172.21.15.73:0/3036770761 shutdown_connections 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 -- 172.21.15.73:0/3036770761 wait complete. 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.461+0000 7f0db703b700 1 Processor -- start 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db703b700 1 -- start start 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db703b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0111830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db703b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0111d70 con 0x7f0db0076c20 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db4dd7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0111830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db4dd7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0111830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53980/0 (socket says 172.21.15.73:53980) 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db4dd7700 1 -- 172.21.15.73:0/2117854176 learned_addr learned my addr 172.21.15.73:0/2117854176 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db4dd7700 1 -- 172.21.15.73:0/2117854176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da40094f0 con 0x7f0db0076c20 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0db4dd7700 1 --2- 172.21.15.73:0/2117854176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0111830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0da400b440 tx=0x7f0da4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.462+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0da40107c0 con 0x7f0db0076c20 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.463+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0da4010db0 con 0x7f0db0076c20 2024-02-10T06:33:54.000 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.463+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db0111f70 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.463+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0da401ad30 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.463+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db0112c10 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.463+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0da402e6c0 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.463+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0da40249c0 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.464+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db005f5c0 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.466+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0da401a430 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.532+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0db0064ad0 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.532+0000 7f0dad7fa700 1 -- 172.21.15.73:0/2117854176 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f0da402eba0 con 0x7f0db0076c20 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.532+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 msgr2=0x7f0db0111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.532+0000 7f0db703b700 1 --2- 172.21.15.73:0/2117854176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0111830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0da400b440 tx=0x7f0da4009250 comp rx=0 tx=0).stop 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.533+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 shutdown_connections 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.533+0000 7f0db703b700 1 --2- 172.21.15.73:0/2117854176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0db0076c20 0x7f0db0111830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:54.001 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.533+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 >> 172.21.15.73:0/2117854176 conn(0x7f0db00fcdd0 msgr2=0x7f0db00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:54.002 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.533+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 shutdown_connections 2024-02-10T06:33:54.002 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:52.533+0000 7f0db703b700 1 -- 172.21.15.73:0/2117854176 wait complete. 2024-02-10T06:33:54.002 INFO:teuthology.orchestra.run.smithi073.stdout:Restarting the monitor... 2024-02-10T06:33:54.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:54 smithi073 systemd[1]: Stopping Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:33:54.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[28690]: 2024-02-10T06:33:54.284+0000 7fb8ed0ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi073 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:33:54.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[28690]: 2024-02-10T06:33:54.284+0000 7fb8ed0ae700 -1 mon.smithi073@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T06:33:54.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:54 smithi073 podman[28939]: 2024-02-10 06:33:54.551083173 +0000 UTC m=+0.379146733 container died 6fca2a65a47e3004c2d2356167d701caa65c50ff7012908117e7f9d9365da68f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T06:33:55.361 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 podman[28939]: 2024-02-10 06:33:55.152245045 +0000 UTC m=+0.980308613 container cleanup 6fca2a65a47e3004c2d2356167d701caa65c50ff7012908117e7f9d9365da68f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:33:55.361 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 bash[28939]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073 2024-02-10T06:33:55.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 podman[28952]: 2024-02-10 06:33:55.361408974 +0000 UTC m=+0.807409615 container remove 6fca2a65a47e3004c2d2356167d701caa65c50ff7012908117e7f9d9365da68f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=quincy-72ce7c1, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T06:33:55.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service: Deactivated successfully. 2024-02-10T06:33:55.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 systemd[1]: Stopped Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:33:55.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 systemd[1]: Starting Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:33:55.996 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 podman[29044]: 2024-02-10 06:33:55.614079388 +0000 UTC m=+0.018507993 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph:quincy 2024-02-10T06:33:56.298 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:55 smithi073 podman[29044]: 2024-02-10 06:33:55.995987813 +0000 UTC m=+0.400416444 container create 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T06:33:56.298 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 podman[29044]: 2024-02-10 06:33:56.20696276 +0000 UTC m=+0.611391358 container init 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 podman[29044]: 2024-02-10 06:33:56.210333849 +0000 UTC m=+0.614762438 container start 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable), process ceph-mon, pid 2 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: pidfile_write: ignore empty --pid-file 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: load: jerasure load: lrc 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: RocksDB version: 6.15.5 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Compile date Feb 9 2024 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: DB SUMMARY 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: DB Session ID: AAQEL9H06DYNSEM20YZI 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:33:56.299 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi073/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi073/store.db: 000010.log size: 97276 ; 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:33:56.300 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.env: 0x55886b0b6a80 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.fs: Posix File System 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.info_log: 0x55886bfc1cc0 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.statistics: (nil) 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.use_fsync: 0 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:33:56.301 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.db_log_dir: 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi073/store.db 2024-02-10T06:33:56.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.write_buffer_manager: 0x55886cd36b40 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:33:56.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.unordered_write: 0 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.row_cache: None 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.wal_filter: None 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:33:56.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.preserve_deletes: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:33:56.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:33:56.306 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:33:56.306 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:33:56.306 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:33:56.306 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:33:56.306 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:33:56.307 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_open_files: -1 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:33:56.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Compression algorithms supported: 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kZSTD supported: 0 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi073/store.db/MANIFEST-000009 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.merge_operator: 2024-02-10T06:33:56.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_filter: None 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55886c016f58) 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_type: 0 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_block_index_type: 0 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_shortening: 1 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:33:56.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: hash_index_allow_collision: 1 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: checksum: 1 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: no_block_cache: 0 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache: 0x55886c049090 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_options: 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: capacity : 536870912 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: num_shard_bits : 4 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: strict_capacity_limit : 0 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_compressed: (nil) 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: persistent_cache: (nil) 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_size: 4096 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_size_deviation: 10 2024-02-10T06:33:56.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_restart_interval: 16 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_block_restart_interval: 1 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: metadata_block_size: 4096 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: partition_filters: 0 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: use_delta_encoding: 1 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: whole_key_filtering: 1 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: verify_compression: 0 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: format_version: 4 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: enable_index_compression: 1 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_align: 0 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression: NoCompression 2024-02-10T06:33:56.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.num_levels: 7 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:33:56.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:33:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:33:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:33:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:33:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:33:56.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:33:56.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.ttl: 2592000 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:33:56.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi073/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546836237441, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546836238272, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92548, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90643, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83358, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707546836, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "7113d16d-77f6-49b2-a498-e65a69d34831", "db_session_id": "AAQEL9H06DYNSEM20YZI"}} 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546836238609, "job": 1, "event": "recovery_finished"} 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi073/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55886bfdd180 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: DB pointer 0x55886c0b2000 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T06:33:56.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** DB Stats ** 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative compaction: 0.00 GB write, 28.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:33:56.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval compaction: 0.00 GB write, 28.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:33:56.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.9 0.00 0.00 1 0.001 0 0 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative compaction: 0.00 GB write, 28.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: starting mon.smithi073 rank 0 at public addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] at bind addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi073 fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???) e1 preinit fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:33:56.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).mds e1 new map 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).mds e1 print_map 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: e1 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: legacy client fscid: -1 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: No filesystems configured 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 bash[29044]: 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 2024-02-10T06:33:56.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 systemd[1]: Started Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:33:56.325 INFO:teuthology.orchestra.run.smithi073.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-10T06:33:56.550 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mon.smithi073 is new leader, mons smithi073 in quorum (ranks 0) 2024-02-10T06:33:56.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: monmap e1: 1 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:33:56.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: fsmap 2024-02-10T06:33:56.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T06:33:56.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:56 smithi073 ceph-mon[29063]: mgrmap e1: no daemons active 2024-02-10T06:33:57.366 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.741+0000 7f8f98d46700 1 Processor -- start 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.741+0000 7f8f98d46700 1 -- start start 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.741+0000 7f8f98d46700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f94105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.741+0000 7f8f98d46700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f941058c0 con 0x7f8f94071480 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9259c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f94105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9259c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f94105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53986/0 (socket says 172.21.15.73:53986) 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9259c700 1 -- 172.21.15.73:0/1372680345 learned_addr learned my addr 172.21.15.73:0/1372680345 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:57.371 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9259c700 1 -- 172.21.15.73:0/1372680345 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94105a00 con 0x7f8f94071480 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9259c700 1 --2- 172.21.15.73:0/1372680345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f94105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8f880098e0 tx=0x7f8f88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d5742254229924 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9159a700 1 -- 172.21.15.73:0/1372680345 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8800b860 con 0x7f8f94071480 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9159a700 1 -- 172.21.15.73:0/1372680345 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8f8800be50 con 0x7f8f94071480 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.742+0000 7f8f9159a700 1 -- 172.21.15.73:0/1372680345 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f88010380 con 0x7f8f94071480 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 -- 172.21.15.73:0/1372680345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 msgr2=0x7f8f94105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 --2- 172.21.15.73:0/1372680345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f94105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8f880098e0 tx=0x7f8f88009250 comp rx=0 tx=0).stop 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 -- 172.21.15.73:0/1372680345 shutdown_connections 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 --2- 172.21.15.73:0/1372680345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f94105380 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 -- 172.21.15.73:0/1372680345 >> 172.21.15.73:0/1372680345 conn(0x7f8f940fcdd0 msgr2=0x7f8f940ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 -- 172.21.15.73:0/1372680345 shutdown_connections 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.743+0000 7f8f98d46700 1 -- 172.21.15.73:0/1372680345 wait complete. 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f98d46700 1 Processor -- start 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f98d46700 1 -- start start 2024-02-10T06:33:57.372 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f98d46700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f941115a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f98d46700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f94111ae0 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f9259c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f941115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f9259c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f941115a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53992/0 (socket says 172.21.15.73:53992) 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f9259c700 1 -- 172.21.15.73:0/188330888 learned_addr learned my addr 172.21.15.73:0/188330888 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f9259c700 1 -- 172.21.15.73:0/188330888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f880094f0 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f9259c700 1 --2- 172.21.15.73:0/188330888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f941115a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f88006b00 tx=0x7f8f88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f880107c0 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f94111ce0 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8f88010db0 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.744+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8801acf0 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.745+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94112980 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.745+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8f88024440 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.745+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8f88019710 con 0x7f8f94071480 2024-02-10T06:33:57.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.745+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9405f5c0 con 0x7f8f94071480 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.748+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8f8801a530 con 0x7f8f94071480 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.813+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8f94106810 con 0x7f8f94071480 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.815+0000 7f8f82ffd700 1 -- 172.21.15.73:0/188330888 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8f88010920 con 0x7f8f94071480 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 msgr2=0x7f8f941115a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 --2- 172.21.15.73:0/188330888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f941115a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f88006b00 tx=0x7f8f88009250 comp rx=0 tx=0).stop 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 shutdown_connections 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 --2- 172.21.15.73:0/188330888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f94071480 0x7f8f941115a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 >> 172.21.15.73:0/188330888 conn(0x7f8f940fcdd0 msgr2=0x7f8f940fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 shutdown_connections 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:56.816+0000 7f8f98d46700 1 -- 172.21.15.73:0/188330888 wait complete. 2024-02-10T06:33:57.374 INFO:teuthology.orchestra.run.smithi073.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-10T06:33:57.375 INFO:teuthology.orchestra.run.smithi073.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:33:57.375 INFO:teuthology.orchestra.run.smithi073.stdout:Creating mgr... 2024-02-10T06:33:57.375 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying port 9283 ... 2024-02-10T06:33:57.552 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mgr.smithi073.vjivfn 2024-02-10T06:33:57.552 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mgr.smithi073.vjivfn.service: Unit ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mgr.smithi073.vjivfn.service not loaded. 2024-02-10T06:33:57.702 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715.target.wants/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mgr.smithi073.vjivfn.service → /etc/systemd/system/ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@.service. 2024-02-10T06:33:58.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:57 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/188330888' entity='client.admin' 2024-02-10T06:33:58.272 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:33:58.273 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-02-10T06:33:58.279 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:33:58.280 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-02-10T06:33:58.280 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-02-10T06:33:58.280 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T06:33:58.288 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:33:58.289 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-02-10T06:33:58.296 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:33:58.296 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-02-10T06:33:58.297 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-02-10T06:33:58.297 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T06:33:58.297 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr to start... 2024-02-10T06:33:58.297 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr... 2024-02-10T06:33:59.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:33:58 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3343735509' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:33:59.456 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:33:59.456 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:33:59.456 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "3c2b87a2-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:33:59.456 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:33:59.457 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "smithi073" 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:33:59.458 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:33:59.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:33:59.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:33:49.429440+0000", 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.751+0000 7f14b0463700 1 Processor -- start 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.751+0000 7f14b0463700 1 -- start start 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.751+0000 7f14b0463700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.751+0000 7f14b0463700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80790b0 con 0x7f14a8078ca0 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ae1ff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:59.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ae1ff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57558/0 (socket says 172.21.15.73:57558) 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ae1ff700 1 -- 172.21.15.73:0/622050390 learned_addr learned my addr 172.21.15.73:0/622050390 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ae1ff700 1 -- 172.21.15.73:0/622050390 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80776f0 con 0x7f14a8078ca0 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ae1ff700 1 --2- 172.21.15.73:0/622050390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a80771b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f14a40098e0 tx=0x7f14a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63eb78a460ce7a24 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ad1fd700 1 -- 172.21.15.73:0/622050390 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14a400b860 con 0x7f14a8078ca0 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ad1fd700 1 -- 172.21.15.73:0/622050390 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f14a400be50 con 0x7f14a8078ca0 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.752+0000 7f14ad1fd700 1 -- 172.21.15.73:0/622050390 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14a4010380 con 0x7f14a8078ca0 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 -- 172.21.15.73:0/622050390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 msgr2=0x7f14a80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 --2- 172.21.15.73:0/622050390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a80771b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f14a40098e0 tx=0x7f14a4009250 comp rx=0 tx=0).stop 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 -- 172.21.15.73:0/622050390 shutdown_connections 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 --2- 172.21.15.73:0/622050390 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a80771b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 -- 172.21.15.73:0/622050390 >> 172.21.15.73:0/622050390 conn(0x7f14a80fded0 msgr2=0x7f14a8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 -- 172.21.15.73:0/622050390 shutdown_connections 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.753+0000 7f14b0463700 1 -- 172.21.15.73:0/622050390 wait complete. 2024-02-10T06:33:59.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14b0463700 1 Processor -- start 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14b0463700 1 -- start start 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14b0463700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a81136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14b0463700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a8113c00 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14ae1ff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a81136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14ae1ff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a81136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57568/0 (socket says 172.21.15.73:57568) 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.754+0000 7f14ae1ff700 1 -- 172.21.15.73:0/3343735509 learned_addr learned my addr 172.21.15.73:0/3343735509 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f14ae1ff700 1 -- 172.21.15.73:0/3343735509 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40094f0 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f14ae1ff700 1 --2- 172.21.15.73:0/3343735509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a81136c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006d40 tx=0x7f14a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14a4010760 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f14a4010d50 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14a4024d20 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a8113e00 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.755+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a8114aa0 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.756+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f14a402e440 con 0x7f14a8078ca0 2024-02-10T06:33:59.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.756+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f14a401a780 con 0x7f14a8078ca0 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.756+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a805f5c0 con 0x7f14a8078ca0 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.758+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f14a4029070 con 0x7f14a8078ca0 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.828+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f14a8108a70 con 0x7f14a8078ca0 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.828+0000 7f149effd700 1 -- 172.21.15.73:0/3343735509 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f14a4010b40 con 0x7f14a8078ca0 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 msgr2=0x7f14a81136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 --2- 172.21.15.73:0/3343735509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a81136c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006d40 tx=0x7f14a4009250 comp rx=0 tx=0).stop 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 shutdown_connections 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 --2- 172.21.15.73:0/3343735509 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14a8078ca0 0x7f14a81136c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 >> 172.21.15.73:0/3343735509 conn(0x7f14a80fded0 msgr2=0x7f14a80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 shutdown_connections 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:33:58.830+0000 7f14b0463700 1 -- 172.21.15.73:0/3343735509 wait complete. 2024-02-10T06:33:59.464 INFO:teuthology.orchestra.run.smithi073.stdout:mgr not available, waiting (1/15)... 2024-02-10T06:34:02.592 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:02 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3070919936' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "3c2b87a2-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:02.637 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "smithi073" 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:34:02.638 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:34:02.646 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:34:02.647 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:02.648 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:33:49.429440+0000", 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.876+0000 7febaef3a700 1 Processor -- start 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.876+0000 7febaef3a700 1 -- start start 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaef3a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8103020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaef3a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba8103560 con 0x7feba8102c10 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaccd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:02.649 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaccd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57582/0 (socket says 172.21.15.73:57582) 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaccd6700 1 -- 172.21.15.73:0/2827351572 learned_addr learned my addr 172.21.15.73:0/2827351572 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaccd6700 1 -- 172.21.15.73:0/2827351572 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba81036a0 con 0x7feba8102c10 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7febaccd6700 1 --2- 172.21.15.73:0/2827351572 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8103020 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0098e0 tx=0x7feb9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6454e33870fad21 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.877+0000 7feba77fe700 1 -- 172.21.15.73:0/2827351572 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb9c00b860 con 0x7feba8102c10 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7feba77fe700 1 -- 172.21.15.73:0/2827351572 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feb9c00be50 con 0x7feba8102c10 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7feba77fe700 1 -- 172.21.15.73:0/2827351572 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb9c010380 con 0x7feba8102c10 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 -- 172.21.15.73:0/2827351572 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 msgr2=0x7feba8103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 --2- 172.21.15.73:0/2827351572 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8103020 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0098e0 tx=0x7feb9c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 -- 172.21.15.73:0/2827351572 shutdown_connections 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 --2- 172.21.15.73:0/2827351572 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8103020 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 -- 172.21.15.73:0/2827351572 >> 172.21.15.73:0/2827351572 conn(0x7feba80fc580 msgr2=0x7feba80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 -- 172.21.15.73:0/2827351572 shutdown_connections 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.878+0000 7febaef3a700 1 -- 172.21.15.73:0/2827351572 wait complete. 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaef3a700 1 Processor -- start 2024-02-10T06:34:02.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaef3a700 1 -- start start 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaef3a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8114560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaef3a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba81014a0 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaccd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8114560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaccd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8114560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57598/0 (socket says 172.21.15.73:57598) 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.879+0000 7febaccd6700 1 -- 172.21.15.73:0/3070919936 learned_addr learned my addr 172.21.15.73:0/3070919936 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7febaccd6700 1 -- 172.21.15.73:0/3070919936 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0094f0 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7febaccd6700 1 --2- 172.21.15.73:0/3070919936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8114560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b440 tx=0x7feb9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb9c0107c0 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feb9c010db0 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba81016a0 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba8114c60 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb9c01ad30 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7feb9c02e6c0 con 0x7feba8102c10 2024-02-10T06:34:02.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.880+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7feb9c024940 con 0x7feba8102c10 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.881+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c0052e0 con 0x7feba8102c10 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.883+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7feb9c01a430 con 0x7feba8102c10 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.954+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feb8c005860 con 0x7feba8102c10 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.954+0000 7feba57fa700 1 -- 172.21.15.73:0/3070919936 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7feb9c024ba0 con 0x7feba8102c10 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 msgr2=0x7feba8114560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 --2- 172.21.15.73:0/3070919936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8114560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b440 tx=0x7feb9c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 shutdown_connections 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 --2- 172.21.15.73:0/3070919936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feba8102c10 0x7feba8114560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 >> 172.21.15.73:0/3070919936 conn(0x7feba80fc580 msgr2=0x7feba80fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 shutdown_connections 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:01.955+0000 7febaef3a700 1 -- 172.21.15.73:0/3070919936 wait complete. 2024-02-10T06:34:02.652 INFO:teuthology.orchestra.run.smithi073.stdout:mgr not available, waiting (2/15)... 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: Activating manager daemon smithi073.vjivfn 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: mgrmap e2: smithi073.vjivfn(active, starting, since 0.00278539s) 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr metadata", "who": "smithi073.vjivfn", "id": "smithi073.vjivfn"}]: dispatch 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: Manager daemon smithi073.vjivfn is now available 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:34:04.655 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/trash_purge_schedule"}]: dispatch 2024-02-10T06:34:04.656 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:04.656 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:04.934 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:04 smithi073 ceph-mon[29063]: from='mgr.14100 172.21.15.73:0/1850289412' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:05.718 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:34:05.718 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:05.718 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "3c2b87a2-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:34:05.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:05.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:34:05.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "smithi073" 2024-02-10T06:34:05.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:05.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-10T06:34:05.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:34:05.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:05.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:34:05.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:34:05.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:34:05.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:34:05.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:34:05.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:34:05.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T06:34:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:34:05.725 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:34:05.725 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:34:05.725 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:34:05.725 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:34:05.725 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:05.725 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:34:05.726 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.726 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:34:05.726 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:05.726 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:33:49.429440+0000", 2024-02-10T06:34:05.726 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:34:05.726 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.026+0000 7f5a2175d700 1 Processor -- start 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.026+0000 7f5a2175d700 1 -- start start 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a2175d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a2175d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c1030d0 con 0x7f5a1c102780 2024-02-10T06:34:05.727 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a1affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:05.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a1affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57680/0 (socket says 172.21.15.73:57680) 2024-02-10T06:34:05.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a1affd700 1 -- 172.21.15.73:0/733631790 learned_addr learned my addr 172.21.15.73:0/733631790 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:05.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a1affd700 1 -- 172.21.15.73:0/733631790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c103210 con 0x7f5a1c102780 2024-02-10T06:34:05.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a1affd700 1 --2- 172.21.15.73:0/733631790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c102b90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5a140098e0 tx=0x7f5a14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ede09290adf43d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:05.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a19ffb700 1 -- 172.21.15.73:0/733631790 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1400b860 con 0x7f5a1c102780 2024-02-10T06:34:05.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a19ffb700 1 -- 172.21.15.73:0/733631790 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5a1400be50 con 0x7f5a1c102780 2024-02-10T06:34:05.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.027+0000 7f5a19ffb700 1 -- 172.21.15.73:0/733631790 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a14010380 con 0x7f5a1c102780 2024-02-10T06:34:05.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 -- 172.21.15.73:0/733631790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 msgr2=0x7f5a1c102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:05.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 --2- 172.21.15.73:0/733631790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c102b90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5a140098e0 tx=0x7f5a14009250 comp rx=0 tx=0).stop 2024-02-10T06:34:05.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 -- 172.21.15.73:0/733631790 shutdown_connections 2024-02-10T06:34:05.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 --2- 172.21.15.73:0/733631790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c102b90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:05.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 -- 172.21.15.73:0/733631790 >> 172.21.15.73:0/733631790 conn(0x7f5a1c0fdf00 msgr2=0x7f5a1c100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:05.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 -- 172.21.15.73:0/733631790 shutdown_connections 2024-02-10T06:34:05.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.028+0000 7f5a2175d700 1 -- 172.21.15.73:0/733631790 wait complete. 2024-02-10T06:34:05.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a2175d700 1 Processor -- start 2024-02-10T06:34:05.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a2175d700 1 -- start start 2024-02-10T06:34:05.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a2175d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:05.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a2175d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c111aa0 con 0x7f5a1c102780 2024-02-10T06:34:05.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a1affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:05.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a1affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57682/0 (socket says 172.21.15.73:57682) 2024-02-10T06:34:05.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a1affd700 1 -- 172.21.15.73:0/3171821424 learned_addr learned my addr 172.21.15.73:0/3171821424 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:05.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a1affd700 1 -- 172.21.15.73:0/3171821424 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140094f0 con 0x7f5a1c102780 2024-02-10T06:34:05.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a1affd700 1 --2- 172.21.15.73:0/3171821424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5a14000c00 tx=0x7f5a14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:05.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a140107c0 con 0x7f5a1c102780 2024-02-10T06:34:05.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.029+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c111ca0 con 0x7f5a1c102780 2024-02-10T06:34:05.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.030+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5a14010db0 con 0x7f5a1c102780 2024-02-10T06:34:05.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.030+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1401ad00 con 0x7f5a1c102780 2024-02-10T06:34:05.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.030+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c112940 con 0x7f5a1c102780 2024-02-10T06:34:05.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.030+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f5a140193c0 con 0x7f5a1c102780 2024-02-10T06:34:05.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.030+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c05f5c0 con 0x7f5a1c102780 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.031+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5a1404a1d0 con 0x7f5a1c102780 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.033+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5a14022930 con 0x7f5a1c102780 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.101+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5a1c04bed0 con 0x7f5a1c102780 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.101+0000 7f5a137fe700 1 -- 172.21.15.73:0/3171821424 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5a14022340 con 0x7f5a1c102780 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 msgr2=0x7f5a1c111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 --2- 172.21.15.73:0/3171821424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5a14000c00 tx=0x7f5a14009250 comp rx=0 tx=0).stop 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 shutdown_connections 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 --2- 172.21.15.73:0/3171821424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a1c102780 0x7f5a1c111560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:05.733 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 >> 172.21.15.73:0/3171821424 conn(0x7f5a1c0fdf00 msgr2=0x7f5a1c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:05.734 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 shutdown_connections 2024-02-10T06:34:05.734 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:05.102+0000 7f5a2175d700 1 -- 172.21.15.73:0/3171821424 wait complete. 2024-02-10T06:34:05.734 INFO:teuthology.orchestra.run.smithi073.stdout:mgr not available, waiting (3/15)... 2024-02-10T06:34:06.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:05 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3171821424' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:34:06.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:05 smithi073 ceph-mon[29063]: mgrmap e3: smithi073.vjivfn(active, since 1.00578s) 2024-02-10T06:34:08.431 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:08 smithi073 ceph-mon[29063]: mgrmap e4: smithi073.vjivfn(active, since 2s) 2024-02-10T06:34:09.201 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:34:09.201 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "3c2b87a2-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "smithi073" 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:09.202 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:34:09.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:33:49.429440+0000", 2024-02-10T06:34:09.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f55a0a70700 1 Processor -- start 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f55a0a70700 1 -- start start 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f55a0a70700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f55a0a70700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c1030c0 con 0x7f559c102770 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f559a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f559a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57690/0 (socket says 172.21.15.73:57690) 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f559a59c700 1 -- 172.21.15.73:0/3440190911 learned_addr learned my addr 172.21.15.73:0/3440190911 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f559a59c700 1 -- 172.21.15.73:0/3440190911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c103200 con 0x7f559c102770 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.169+0000 7f559a59c700 1 --2- 172.21.15.73:0/3440190911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c102b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f558c0098e0 tx=0x7f558c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8e3ca60a27df85 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f559959a700 1 -- 172.21.15.73:0/3440190911 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558c00b860 con 0x7f559c102770 2024-02-10T06:34:09.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f559959a700 1 -- 172.21.15.73:0/3440190911 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f558c00be50 con 0x7f559c102770 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f559959a700 1 -- 172.21.15.73:0/3440190911 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558c010380 con 0x7f559c102770 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 -- 172.21.15.73:0/3440190911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 msgr2=0x7f559c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 --2- 172.21.15.73:0/3440190911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c102b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f558c0098e0 tx=0x7f558c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 -- 172.21.15.73:0/3440190911 shutdown_connections 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 --2- 172.21.15.73:0/3440190911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c102b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 -- 172.21.15.73:0/3440190911 >> 172.21.15.73:0/3440190911 conn(0x7f559c0fded0 msgr2=0x7f559c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 -- 172.21.15.73:0/3440190911 shutdown_connections 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.170+0000 7f55a0a70700 1 -- 172.21.15.73:0/3440190911 wait complete. 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f55a0a70700 1 Processor -- start 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f55a0a70700 1 -- start start 2024-02-10T06:34:09.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f55a0a70700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f55a0a70700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c111aa0 con 0x7f559c102770 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f559a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f559a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57698/0 (socket says 172.21.15.73:57698) 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f559a59c700 1 -- 172.21.15.73:0/2973191208 learned_addr learned my addr 172.21.15.73:0/2973191208 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f559a59c700 1 -- 172.21.15.73:0/2973191208 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0094f0 con 0x7f559c102770 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.171+0000 7f559a59c700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c111560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f558c000c00 tx=0x7f558c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558c0107c0 con 0x7f559c102770 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c111ca0 con 0x7f559c102770 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f558c010db0 con 0x7f559c102770 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558c01ad00 con 0x7f559c102770 2024-02-10T06:34:09.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c112940 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f558c0193c0 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f558affd700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f5584036740 0x7f5584038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.172+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c05f5c0 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.173+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f558c04a780 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.173+0000 7f5599d9b700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f5584036740 0x7f5584038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.173+0000 7f5599d9b700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f5584036740 0x7f5584038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5590009990 tx=0x7f5590006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.178+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f558c019670 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.430+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f559c04bed0 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.431+0000 7f558affd700 1 -- 172.21.15.73:0/2973191208 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f558c031020 con 0x7f559c102770 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f5584036740 msgr2=0x7f5584038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f5584036740 0x7f5584038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5590009990 tx=0x7f5590006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:09.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 msgr2=0x7f559c111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c111560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f558c000c00 tx=0x7f558c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 shutdown_connections 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f5584036740 0x7f5584038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 --2- 172.21.15.73:0/2973191208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f559c102770 0x7f559c111560 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 >> 172.21.15.73:0/2973191208 conn(0x7f559c0fded0 msgr2=0x7f559c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 shutdown_connections 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:08.433+0000 7f55a0a70700 1 -- 172.21.15.73:0/2973191208 wait complete. 2024-02-10T06:34:09.210 INFO:teuthology.orchestra.run.smithi073.stdout:mgr is available 2024-02-10T06:34:09.523 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:09 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2973191208' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:34:10.335 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:10 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3978847474' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T06:34:10.349 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:34:10.349 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout fsid = 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-10T06:34:10.350 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.623+0000 7f12670fa700 1 Processor -- start 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.623+0000 7f12670fa700 1 -- start start 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f12670fa700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f12670fa700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12601058a0 con 0x7f12600714a0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f1264e96700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f1264e96700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48232/0 (socket says 172.21.15.73:48232) 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f1264e96700 1 -- 172.21.15.73:0/504227770 learned_addr learned my addr 172.21.15.73:0/504227770 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f1264e96700 1 -- 172.21.15.73:0/504227770 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12601059e0 con 0x7f12600714a0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f1264e96700 1 --2- 172.21.15.73:0/504227770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260105360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f12540098e0 tx=0x7f1254009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f7750b26704290 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f125f7fe700 1 -- 172.21.15.73:0/504227770 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f125400b860 con 0x7f12600714a0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f125f7fe700 1 -- 172.21.15.73:0/504227770 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f125400be50 con 0x7f12600714a0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.624+0000 7f125f7fe700 1 -- 172.21.15.73:0/504227770 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1254010380 con 0x7f12600714a0 2024-02-10T06:34:10.351 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 -- 172.21.15.73:0/504227770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 msgr2=0x7f1260105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 --2- 172.21.15.73:0/504227770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260105360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f12540098e0 tx=0x7f1254009250 comp rx=0 tx=0).stop 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 -- 172.21.15.73:0/504227770 shutdown_connections 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 --2- 172.21.15.73:0/504227770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260105360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 -- 172.21.15.73:0/504227770 >> 172.21.15.73:0/504227770 conn(0x7f12600fcdd0 msgr2=0x7f12600ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 -- 172.21.15.73:0/504227770 shutdown_connections 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.625+0000 7f12670fa700 1 -- 172.21.15.73:0/504227770 wait complete. 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f12670fa700 1 Processor -- start 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f12670fa700 1 -- start start 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f12670fa700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f12670fa700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260111ac0 con 0x7f12600714a0 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f1264e96700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:10.352 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f1264e96700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48234/0 (socket says 172.21.15.73:48234) 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f1264e96700 1 -- 172.21.15.73:0/3978847474 learned_addr learned my addr 172.21.15.73:0/3978847474 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.626+0000 7f1264e96700 1 -- 172.21.15.73:0/3978847474 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12540094f0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.627+0000 7f1264e96700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260111580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1254006b00 tx=0x7f1254009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.627+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12540107c0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.627+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1254010db0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.627+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1260111cc0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.627+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f125401acf0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.627+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1260112960 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.628+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f12540193c0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.628+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126005f5c0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.628+0000 7f125d7fa700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1250036790 0x7f1250038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.628+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f125404a6c0 con 0x7f12600714a0 2024-02-10T06:34:10.353 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.628+0000 7f125ffff700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1250036790 0x7f1250038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.628+0000 7f125ffff700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1250036790 0x7f1250038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f124c009990 tx=0x7f124c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.633+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1254019670 con 0x7f12600714a0 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.812+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f126004bed0 con 0x7f12600714a0 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.813+0000 7f125d7fa700 1 -- 172.21.15.73:0/3978847474 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f1254022340 con 0x7f12600714a0 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1250036790 msgr2=0x7f1250038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1250036790 0x7f1250038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f124c009990 tx=0x7f124c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 msgr2=0x7f1260111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260111580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1254006b00 tx=0x7f1254009250 comp rx=0 tx=0).stop 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 shutdown_connections 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1250036790 0x7f1250038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 --2- 172.21.15.73:0/3978847474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12600714a0 0x7f1260111580 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 >> 172.21.15.73:0/3978847474 conn(0x7f12600fcdd0 msgr2=0x7f12600fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 shutdown_connections 2024-02-10T06:34:10.354 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:09.816+0000 7f12670fa700 1 -- 172.21.15.73:0/3978847474 wait complete. 2024-02-10T06:34:10.355 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling cephadm module... 2024-02-10T06:34:11.583 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:11 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1148410221' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T06:34:11.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cfe7da700 1 Processor -- start 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cfe7da700 1 -- start start 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cfe7da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cfe7da700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80790f0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf80771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48246/0 (socket says 172.21.15.73:48246) 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.750+0000 7f1cf7fff700 1 -- 172.21.15.73:0/54237167 learned_addr learned my addr 172.21.15.73:0/54237167 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cf7fff700 1 -- 172.21.15.73:0/54237167 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8077730 con 0x7f1cf8078ce0 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cf7fff700 1 --2- 172.21.15.73:0/54237167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf80771f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098e0 tx=0x7f1ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da858aa2808eab1c server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cf6ffd700 1 -- 172.21.15.73:0/54237167 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce800b860 con 0x7f1cf8078ce0 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cf6ffd700 1 -- 172.21.15.73:0/54237167 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1ce800be50 con 0x7f1cf8078ce0 2024-02-10T06:34:11.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cf6ffd700 1 -- 172.21.15.73:0/54237167 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce8010380 con 0x7f1cf8078ce0 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cfe7da700 1 -- 172.21.15.73:0/54237167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 msgr2=0x7f1cf80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.751+0000 7f1cfe7da700 1 --2- 172.21.15.73:0/54237167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf80771f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098e0 tx=0x7f1ce8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 -- 172.21.15.73:0/54237167 shutdown_connections 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 --2- 172.21.15.73:0/54237167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf80771f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 -- 172.21.15.73:0/54237167 >> 172.21.15.73:0/54237167 conn(0x7f1cf80fded0 msgr2=0x7f1cf8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 -- 172.21.15.73:0/54237167 shutdown_connections 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 -- 172.21.15.73:0/54237167 wait complete. 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 Processor -- start 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 -- start start 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf8113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.752+0000 7f1cfe7da700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8113d60 con 0x7f1cf8078ce0 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf8113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf8113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48250/0 (socket says 172.21.15.73:48250) 2024-02-10T06:34:11.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf7fff700 1 -- 172.21.15.73:0/1148410221 learned_addr learned my addr 172.21.15.73:0/1148410221 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf7fff700 1 -- 172.21.15.73:0/1148410221 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80094f0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf7fff700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf8113820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800b440 tx=0x7f1ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce8010760 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1ce8010d50 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce801adc0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf8113f60 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.753+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf8114c00 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.754+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f1ce80193c0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.754+0000 7f1cf4ff9700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1ce0036790 0x7f1ce0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.754+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf805f5c0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.754+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1ce8049fb0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.754+0000 7f1cf77fe700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1ce0036790 0x7f1ce0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.755+0000 7f1cf77fe700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1ce0036790 0x7f1ce0038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009990 tx=0x7f1cec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:11.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.759+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1ce8023940 con 0x7f1cf8078ce0 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:10.991+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f1cf804bed0 con 0x7f1cf8078ce0 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.252+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1ce802c350 con 0x7f1cf8078ce0 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.253+0000 7f1cf4ff9700 1 -- 172.21.15.73:0/1148410221 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f1ce8019a30 con 0x7f1cf8078ce0 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1ce0036790 msgr2=0x7f1ce0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1ce0036790 0x7f1ce0038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009990 tx=0x7f1cec006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 msgr2=0x7f1cf8113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf8113820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800b440 tx=0x7f1ce8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 shutdown_connections 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f1ce0036790 0x7f1ce0038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 --2- 172.21.15.73:0/1148410221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cf8078ce0 0x7f1cf8113820 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 >> 172.21.15.73:0/1148410221 conn(0x7f1cf80fded0 msgr2=0x7f1cf80ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 shutdown_connections 2024-02-10T06:34:11.904 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:11.255+0000 7f1cfe7da700 1 -- 172.21.15.73:0/1148410221 wait complete. 2024-02-10T06:34:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:12 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1148410221' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T06:34:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:12 smithi073 ceph-mon[29063]: mgrmap e5: smithi073.vjivfn(active, since 6s) 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "active_name": "smithi073.vjivfn", 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60eb77700 1 Processor -- start 2024-02-10T06:34:13.183 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60eb77700 1 -- start start 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60eb77700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60eb77700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6081030c0 con 0x7fb608102770 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60c913700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60c913700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48276/0 (socket says 172.21.15.73:48276) 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60c913700 1 -- 172.21.15.73:0/1698730892 learned_addr learned my addr 172.21.15.73:0/1698730892 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.307+0000 7fb60c913700 1 -- 172.21.15.73:0/1698730892 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb608103200 con 0x7fb608102770 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60c913700 1 --2- 172.21.15.73:0/1698730892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0098e0 tx=0x7fb5fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d91b500bbc2a1c server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb6077fe700 1 -- 172.21.15.73:0/1698730892 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5fc00b860 con 0x7fb608102770 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb6077fe700 1 -- 172.21.15.73:0/1698730892 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb5fc00be50 con 0x7fb608102770 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb6077fe700 1 -- 172.21.15.73:0/1698730892 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5fc010380 con 0x7fb608102770 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60eb77700 1 -- 172.21.15.73:0/1698730892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 msgr2=0x7fb608102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60eb77700 1 --2- 172.21.15.73:0/1698730892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0098e0 tx=0x7fb5fc009250 comp rx=0 tx=0).stop 2024-02-10T06:34:13.184 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60eb77700 1 -- 172.21.15.73:0/1698730892 shutdown_connections 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60eb77700 1 --2- 172.21.15.73:0/1698730892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60eb77700 1 -- 172.21.15.73:0/1698730892 >> 172.21.15.73:0/1698730892 conn(0x7fb6080fded0 msgr2=0x7fb608100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.308+0000 7fb60eb77700 1 -- 172.21.15.73:0/1698730892 shutdown_connections 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60eb77700 1 -- 172.21.15.73:0/1698730892 wait complete. 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60eb77700 1 Processor -- start 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60eb77700 1 -- start start 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60eb77700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60eb77700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb608077190 con 0x7fb608102770 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60c913700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60c913700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48284/0 (socket says 172.21.15.73:48284) 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.309+0000 7fb60c913700 1 -- 172.21.15.73:0/3041401575 learned_addr learned my addr 172.21.15.73:0/3041401575 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb60c913700 1 -- 172.21.15.73:0/3041401575 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0094f0 con 0x7fb608102770 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb60c913700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b440 tx=0x7fb5fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:13.185 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5fc010760 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb5fc010d50 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5fc024d80 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb608077390 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.310+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb608077820 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.311+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb5fc01a430 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.311+0000 7fb6057fa700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 0x7fb5f8038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.311+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb5fc04a690 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.311+0000 7fb607fff700 1 -- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 msgr2=0x7fb5f8038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.311+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60804bed0 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.311+0000 7fb607fff700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 0x7fb5f8038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.316+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb5fc01a730 con 0x7fb608102770 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.512+0000 7fb607fff700 1 -- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 msgr2=0x7fb5f8038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.512+0000 7fb607fff700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 0x7fb5f8038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:34:13.186 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.559+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb608077ec0 con 0x7fb608102770 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.559+0000 7fb6057fa700 1 -- 172.21.15.73:0/3041401575 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb5fc018350 con 0x7fb608102770 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 msgr2=0x7fb5f8038bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 0x7fb5f8038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 msgr2=0x7fb608078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b440 tx=0x7fb5fc009250 comp rx=0 tx=0).stop 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 shutdown_connections 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7fb5f8036740 0x7fb5f8038bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 --2- 172.21.15.73:0/3041401575 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb608102770 0x7fb608078d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 >> 172.21.15.73:0/3041401575 conn(0x7fb6080fded0 msgr2=0x7fb6080fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 shutdown_connections 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:12.561+0000 7fb60eb77700 1 -- 172.21.15.73:0/3041401575 wait complete. 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for the mgr to restart... 2024-02-10T06:34:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr epoch 5... 2024-02-10T06:34:13.352 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:13 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3041401575' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T06:34:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: Active manager daemon smithi073.vjivfn restarted 2024-02-10T06:34:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: Activating manager daemon smithi073.vjivfn 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: mgrmap e6: smithi073.vjivfn(active, starting, since 0.00349631s) 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr metadata", "who": "smithi073.vjivfn", "id": "smithi073.vjivfn"}]: dispatch 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: Manager daemon smithi073.vjivfn is now available 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:18.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:18.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:17 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:19.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:18 smithi073 ceph-mon[29063]: Found migration_current of "None". Setting to last migration. 2024-02-10T06:34:19.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:18 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:34:19.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:18 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/trash_purge_schedule"}]: dispatch 2024-02-10T06:34:19.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:18 smithi073 ceph-mon[29063]: mgrmap e7: smithi073.vjivfn(active, since 1.00551s) 2024-02-10T06:34:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-10T06:34:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T06:34:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.641+0000 7f4d9396a700 1 Processor -- start 2024-02-10T06:34:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.641+0000 7f4d9396a700 1 -- start start 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.641+0000 7f4d9396a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.641+0000 7f4d9396a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c1030c0 con 0x7f4d8c102770 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d91706700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d91706700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48298/0 (socket says 172.21.15.73:48298) 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d91706700 1 -- 172.21.15.73:0/2269111416 learned_addr learned my addr 172.21.15.73:0/2269111416 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d91706700 1 -- 172.21.15.73:0/2269111416 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c103200 con 0x7f4d8c102770 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d91706700 1 --2- 172.21.15.73:0/2269111416 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c102b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0098e0 tx=0x7f4d7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5136eff296abcd80 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d83fff700 1 -- 172.21.15.73:0/2269111416 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c00b860 con 0x7f4d8c102770 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d83fff700 1 -- 172.21.15.73:0/2269111416 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4d7c00be50 con 0x7f4d8c102770 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.642+0000 7f4d83fff700 1 -- 172.21.15.73:0/2269111416 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c010380 con 0x7f4d8c102770 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.643+0000 7f4d9396a700 1 -- 172.21.15.73:0/2269111416 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 msgr2=0x7f4d8c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.643+0000 7f4d9396a700 1 --2- 172.21.15.73:0/2269111416 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c102b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0098e0 tx=0x7f4d7c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.643+0000 7f4d9396a700 1 -- 172.21.15.73:0/2269111416 shutdown_connections 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.643+0000 7f4d9396a700 1 --2- 172.21.15.73:0/2269111416 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c102b80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.643+0000 7f4d9396a700 1 -- 172.21.15.73:0/2269111416 >> 172.21.15.73:0/2269111416 conn(0x7f4d8c0fded0 msgr2=0x7f4d8c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.643+0000 7f4d9396a700 1 -- 172.21.15.73:0/2269111416 shutdown_connections 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d9396a700 1 -- 172.21.15.73:0/2269111416 wait complete. 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d9396a700 1 Processor -- start 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d9396a700 1 -- start start 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d9396a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d9396a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c111a40 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d91706700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d91706700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48300/0 (socket says 172.21.15.73:48300) 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.644+0000 7f4d91706700 1 -- 172.21.15.73:0/3483853855 learned_addr learned my addr 172.21.15.73:0/3483853855 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d91706700 1 -- 172.21.15.73:0/3483853855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c0094f0 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d91706700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c111500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00b440 tx=0x7f4d7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c010760 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4d7c010d50 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c111c40 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c024d80 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.645+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c1128e0 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.646+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4d7c01a430 con 0x7f4d8c102770 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.646+0000 7f4d81ffb700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.646+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 --> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d8c05f5c0 con 0x7f4d78036740 2024-02-10T06:34:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.646+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4d7c04a6c0 con 0x7f4d8c102770 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.646+0000 7f4d90f05700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 msgr2=0x7f4d78038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.646+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.847+0000 7f4d90f05700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 msgr2=0x7f4d78038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:13.847+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:14.248+0000 7f4d90f05700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 msgr2=0x7f4d78038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:14.248+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:15.049+0000 7f4d90f05700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 msgr2=0x7f4d78038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:15.049+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:16.650+0000 7f4d90f05700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 msgr2=0x7f4d78038bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/194898351 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:16.650+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:17.721+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f4d7c02ad60 con 0x7f4d8c102770 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:17.721+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 msgr2=0x7f4d78038bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:34:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:17.721+0000 7f4d81ffb700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.722+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f4d7c04c100 con 0x7f4d8c102770 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.722+0000 7f4d81ffb700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4d78039f00 0x7f4d7803c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.722+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d8c05f5c0 con 0x7f4d78039f00 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.723+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4d78039f00 0x7f4d7803c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.723+0000 7f4d90f05700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4d78039f00 0x7f4d7803c2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4d880039d0 tx=0x7f4d88009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.724+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f4d8c05f5c0 con 0x7f4d78039f00 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4d8c1120d0 con 0x7f4d78039f00 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d81ffb700 1 -- 172.21.15.73:0/3483853855 <== mgr.14120 v2:172.21.15.73:6800/1157895579 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4d8c1120d0 con 0x7f4d78039f00 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4d78039f00 msgr2=0x7f4d7803c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4d78039f00 0x7f4d7803c2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4d880039d0 tx=0x7f4d88009350 comp rx=0 tx=0).stop 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 msgr2=0x7f4d8c111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c111500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00b440 tx=0x7f4d7c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 shutdown_connections 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4d78039f00 0x7f4d7803c2e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:19.321 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d8c102770 0x7f4d8c111500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:19.322 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 --2- 172.21.15.73:0/3483853855 >> [v2:172.21.15.73:6800/194898351,v1:172.21.15.73:6801/194898351] conn(0x7f4d78036740 0x7f4d78038bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:19.322 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.730+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 >> 172.21.15.73:0/3483853855 conn(0x7f4d8c0fded0 msgr2=0x7f4d8c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:19.322 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.731+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 shutdown_connections 2024-02-10T06:34:19.322 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:18.731+0000 7f4d9396a700 1 -- 172.21.15.73:0/3483853855 wait complete. 2024-02-10T06:34:19.322 INFO:teuthology.orchestra.run.smithi073.stdout:mgr epoch 5 is available 2024-02-10T06:34:19.322 INFO:teuthology.orchestra.run.smithi073.stdout:Setting orchestrator backend to cephadm... 2024-02-10T06:34:19.936 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:19 smithi073 ceph-mon[29063]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T06:34:19.936 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:19 smithi073 ceph-mon[29063]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T06:34:19.936 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:19 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:19.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:19 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.727+0000 7f710b483700 1 Processor -- start 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710b483700 1 -- start start 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710b483700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710b483700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71041030c0 con 0x7f7104102770 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710921f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710921f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57452/0 (socket says 172.21.15.73:57452) 2024-02-10T06:34:20.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710921f700 1 -- 172.21.15.73:0/858769603 learned_addr learned my addr 172.21.15.73:0/858769603 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710921f700 1 -- 172.21.15.73:0/858769603 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7104103200 con 0x7f7104102770 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.728+0000 7f710921f700 1 --2- 172.21.15.73:0/858769603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104102b80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70f40098e0 tx=0x7f70f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2899514130e66cbb server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f70fbfff700 1 -- 172.21.15.73:0/858769603 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f400b860 con 0x7f7104102770 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f70fbfff700 1 -- 172.21.15.73:0/858769603 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f70f400be50 con 0x7f7104102770 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f70fbfff700 1 -- 172.21.15.73:0/858769603 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f4010380 con 0x7f7104102770 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 -- 172.21.15.73:0/858769603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 msgr2=0x7f7104102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 --2- 172.21.15.73:0/858769603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104102b80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70f40098e0 tx=0x7f70f4009250 comp rx=0 tx=0).stop 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 -- 172.21.15.73:0/858769603 shutdown_connections 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 --2- 172.21.15.73:0/858769603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104102b80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:20.556 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 -- 172.21.15.73:0/858769603 >> 172.21.15.73:0/858769603 conn(0x7f71040fded0 msgr2=0x7f7104100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 -- 172.21.15.73:0/858769603 shutdown_connections 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.729+0000 7f710b483700 1 -- 172.21.15.73:0/858769603 wait complete. 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710b483700 1 Processor -- start 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710b483700 1 -- start start 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710b483700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710b483700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104111ad0 con 0x7f7104102770 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710921f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710921f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57462/0 (socket says 172.21.15.73:57462) 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710921f700 1 -- 172.21.15.73:0/81349309 learned_addr learned my addr 172.21.15.73:0/81349309 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710921f700 1 -- 172.21.15.73:0/81349309 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f40094f0 con 0x7f7104102770 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f710921f700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104111590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f70f4000c00 tx=0x7f70f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.730+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f40107c0 con 0x7f7104102770 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.731+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7104111cd0 con 0x7f7104102770 2024-02-10T06:34:20.557 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.731+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f70f4010db0 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.731+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f401ad00 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.731+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7104112970 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.731+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f70f40193c0 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.732+0000 7f70f9ffb700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f70f00366b0 0x7f70f0038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.732+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710405f5c0 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.732+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f70f404a460 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.732+0000 7f7108a1e700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f70f00366b0 0x7f70f0038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.732+0000 7f7108a1e700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f70f00366b0 0x7f70f0038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7100009990 tx=0x7f7100006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.737+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f70f4022930 con 0x7f7104102770 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.934+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f71041001b0 con 0x7f70f00366b0 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.941+0000 7f70f9ffb700 1 -- 172.21.15.73:0/81349309 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f71041001b0 con 0x7f70f00366b0 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f70f00366b0 msgr2=0x7f70f0038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:20.558 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f70f00366b0 0x7f70f0038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7100009990 tx=0x7f7100006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 msgr2=0x7f7104111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104111590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f70f4000c00 tx=0x7f70f4009250 comp rx=0 tx=0).stop 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 shutdown_connections 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f70f00366b0 0x7f70f0038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 --2- 172.21.15.73:0/81349309 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7104102770 0x7f7104111590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 >> 172.21.15.73:0/81349309 conn(0x7f71040fded0 msgr2=0x7f71040ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 shutdown_connections 2024-02-10T06:34:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:19.943+0000 7f710b483700 1 -- 172.21.15.73:0/81349309 wait complete. 2024-02-10T06:34:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: [10/Feb/2024:06:34:19] ENGINE Bus STARTING 2024-02-10T06:34:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: [10/Feb/2024:06:34:19] ENGINE Serving on https://172.21.15.73:7150 2024-02-10T06:34:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: [10/Feb/2024:06:34:19] ENGINE Bus STARTED 2024-02-10T06:34:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:21.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:20 smithi073 ceph-mon[29063]: mgrmap e8: smithi073.vjivfn(active, since 2s) 2024-02-10T06:34:21.728 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fae879700 1 Processor -- start 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fae879700 1 -- start start 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fae879700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fae879700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa8101fc0 con 0x7f4fa8101670 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57472/0 (socket says 172.21.15.73:57472) 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa7fff700 1 -- 172.21.15.73:0/1675161650 learned_addr learned my addr 172.21.15.73:0/1675161650 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa7fff700 1 -- 172.21.15.73:0/1675161650 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa8102100 con 0x7f4fa8101670 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa7fff700 1 --2- 172.21.15.73:0/1675161650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8101a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4f980098e0 tx=0x7f4f98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb769d69c2da40a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa6ffd700 1 -- 172.21.15.73:0/1675161650 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f9800b860 con 0x7f4fa8101670 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.958+0000 7f4fa6ffd700 1 -- 172.21.15.73:0/1675161650 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4f9800be50 con 0x7f4fa8101670 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fa6ffd700 1 -- 172.21.15.73:0/1675161650 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f98010380 con 0x7f4fa8101670 2024-02-10T06:34:21.729 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 -- 172.21.15.73:0/1675161650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 msgr2=0x7f4fa8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 --2- 172.21.15.73:0/1675161650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8101a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4f980098e0 tx=0x7f4f98009250 comp rx=0 tx=0).stop 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 -- 172.21.15.73:0/1675161650 shutdown_connections 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 --2- 172.21.15.73:0/1675161650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8101a80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 -- 172.21.15.73:0/1675161650 >> 172.21.15.73:0/1675161650 conn(0x7f4fa80fcdd0 msgr2=0x7f4fa80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 -- 172.21.15.73:0/1675161650 shutdown_connections 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.959+0000 7f4fae879700 1 -- 172.21.15.73:0/1675161650 wait complete. 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fae879700 1 Processor -- start 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fae879700 1 -- start start 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fae879700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8111510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fae879700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa8111a50 con 0x7f4fa8101670 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fa7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8111510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fa7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8111510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57474/0 (socket says 172.21.15.73:57474) 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fa7fff700 1 -- 172.21.15.73:0/4220567606 learned_addr learned my addr 172.21.15.73:0/4220567606 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fa7fff700 1 -- 172.21.15.73:0/4220567606 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980094f0 con 0x7f4fa8101670 2024-02-10T06:34:21.730 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fa7fff700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8111510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800b440 tx=0x7f4f98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.960+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f98010760 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.961+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4f98010d50 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.961+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa8111c50 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.961+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f98024d80 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.961+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa81128f0 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.961+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4f9801a430 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.962+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa805f5c0 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.962+0000 7f4fa4ff9700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4f900367e0 0x7f4f90038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.962+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4f9804a700 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.962+0000 7f4fa77fe700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4f900367e0 0x7f4f90038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.962+0000 7f4fa77fe700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4f900367e0 0x7f4f90038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009990 tx=0x7f4f9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:20.966+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4f98018530 con 0x7f4fa8101670 2024-02-10T06:34:21.731 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.150+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fa80fe280 con 0x7f4f900367e0 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.151+0000 7f4fa4ff9700 1 -- 172.21.15.73:0/4220567606 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4fa80fe280 con 0x7f4f900367e0 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4f900367e0 msgr2=0x7f4f90038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4f900367e0 0x7f4f90038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009990 tx=0x7f4f9c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 msgr2=0x7f4fa8111510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8111510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800b440 tx=0x7f4f98009250 comp rx=0 tx=0).stop 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 shutdown_connections 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f4f900367e0 0x7f4f90038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 --2- 172.21.15.73:0/4220567606 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4fa8101670 0x7f4fa8111510 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 >> 172.21.15.73:0/4220567606 conn(0x7f4fa80fcdd0 msgr2=0x7f4fa80fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 shutdown_connections 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:21.153+0000 7f4fae879700 1 -- 172.21.15.73:0/4220567606 wait complete. 2024-02-10T06:34:21.732 INFO:teuthology.orchestra.run.smithi073.stdout:Generating ssh key... 2024-02-10T06:34:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:22 smithi073 ceph-mon[29063]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:23.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13b67d0700 1 Processor -- start 2024-02-10T06:34:23.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13b67d0700 1 -- start start 2024-02-10T06:34:23.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13b67d0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:23.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13b67d0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b01058c0 con 0x7f13b0076c20 2024-02-10T06:34:23.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13affff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:23.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13affff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57486/0 (socket says 172.21.15.73:57486) 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13affff700 1 -- 172.21.15.73:0/560296688 learned_addr learned my addr 172.21.15.73:0/560296688 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.185+0000 7f13affff700 1 -- 172.21.15.73:0/560296688 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0105a00 con 0x7f13b0076c20 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13affff700 1 --2- 172.21.15.73:0/560296688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0105380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f13a00098e0 tx=0x7f13a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae3d1a69a25dc2b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13aeffd700 1 -- 172.21.15.73:0/560296688 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a000b860 con 0x7f13b0076c20 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13aeffd700 1 -- 172.21.15.73:0/560296688 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f13a000be50 con 0x7f13b0076c20 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13aeffd700 1 -- 172.21.15.73:0/560296688 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a0010380 con 0x7f13b0076c20 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 -- 172.21.15.73:0/560296688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 msgr2=0x7f13b0105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 --2- 172.21.15.73:0/560296688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0105380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f13a00098e0 tx=0x7f13a0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 -- 172.21.15.73:0/560296688 shutdown_connections 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 --2- 172.21.15.73:0/560296688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0105380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 -- 172.21.15.73:0/560296688 >> 172.21.15.73:0/560296688 conn(0x7f13b00fcdd0 msgr2=0x7f13b00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:23.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 -- 172.21.15.73:0/560296688 shutdown_connections 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.186+0000 7f13b67d0700 1 -- 172.21.15.73:0/560296688 wait complete. 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13b67d0700 1 Processor -- start 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13b67d0700 1 -- start start 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13b67d0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0111830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13b67d0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b0111d70 con 0x7f13b0076c20 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13affff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0111830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13affff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0111830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57500/0 (socket says 172.21.15.73:57500) 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.187+0000 7f13affff700 1 -- 172.21.15.73:0/89394736 learned_addr learned my addr 172.21.15.73:0/89394736 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13affff700 1 -- 172.21.15.73:0/89394736 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00094f0 con 0x7f13b0076c20 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13affff700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0111830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f13a000b440 tx=0x7f13a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a00107c0 con 0x7f13b0076c20 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f13a0010db0 con 0x7f13b0076c20 2024-02-10T06:34:23.397 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b0111f70 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a001ad30 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.188+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b0112c10 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.189+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f13a002e440 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.189+0000 7f13acff9700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f1398036790 0x7f1398038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.189+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f13a004acb0 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.189+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b005f5c0 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.189+0000 7f13af7fe700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f1398036790 0x7f1398038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.189+0000 7f13af7fe700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f1398036790 0x7f1398038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009990 tx=0x7f13a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.194+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f13a00243f0 con 0x7f13b0076c20 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.376+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f13b0106880 con 0x7f1398036790 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.791+0000 7f13acff9700 1 -- 172.21.15.73:0/89394736 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13b0106880 con 0x7f1398036790 2024-02-10T06:34:23.398 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f1398036790 msgr2=0x7f1398038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f1398036790 0x7f1398038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009990 tx=0x7f13a4006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 msgr2=0x7f13b0111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0111830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f13a000b440 tx=0x7f13a0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 shutdown_connections 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f1398036790 0x7f1398038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 --2- 172.21.15.73:0/89394736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13b0076c20 0x7f13b0111830 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 >> 172.21.15.73:0/89394736 conn(0x7f13b00fcdd0 msgr2=0x7f13b00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 shutdown_connections 2024-02-10T06:34:23.399 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:22.793+0000 7f13b67d0700 1 -- 172.21.15.73:0/89394736 wait complete. 2024-02-10T06:34:24.037 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:23 smithi073 ceph-mon[29063]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:24.038 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:23 smithi073 ceph-mon[29063]: Generating ssh key... 2024-02-10T06:34:24.038 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:23 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:24.038 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:23 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:24.670 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRGpo10GKY2Rwb2E6DI1/+giSgvoiDMBtbYYue7vgUtghsB6LXOMRyDU4fG8zSLod/8jt2/IrupU3gVVLiU/3+GROafpq/W2B/pXHOmFAaW1Df4DGajELTuyqCE8SqTcLOrheXTy/s/ZUL1y4rc1hN38i975ZaBW68EvAk8H2TeLbWaSOslcBjm5a7zKWuE9TnbXXqLNZzZuAk70l/4dUpT5KCiXFUe5qXbDFKPbxq+xEV64fwDF+AXseLel4lgKvXfyzH5ADT6tdIOkQ7pV40yicMtbX3EQq2a1elWuSuQKZBaqHJJo6fl/gcuk/2TqUmrvpTbKMf2V2EGi4d5hsZv1pN1Q3MvObWRpx8Zf9/iTr9nYYmniK6QwvMI3vPS0btZi0t8K1lSvDalSBWT8kKtDff7iKO+2vN4AVE6qsnquwiKebbm+iRnEnqTZRHMgL5/gorRIraSla+yZiRFKv/WXEj5LiojoM0W2GtYyEFNP9yDenvZk4Nc9UFeP05kFU= ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.841+0000 7f802fd60700 1 Processor -- start 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.841+0000 7f802fd60700 1 -- start start 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802fd60700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802fd60700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80281031b0 con 0x7f8028071440 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802dafc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802dafc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57514/0 (socket says 172.21.15.73:57514) 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802dafc700 1 -- 172.21.15.73:0/740619682 learned_addr learned my addr 172.21.15.73:0/740619682 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802dafc700 1 -- 172.21.15.73:0/740619682 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8028107d00 con 0x7f8028071440 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802dafc700 1 --2- 172.21.15.73:0/740619682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281077c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8020009e50 tx=0x7f8020009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ced676f5890613e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.842+0000 7f802cafa700 1 -- 172.21.15.73:0/740619682 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80200092e0 con 0x7f8028071440 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802cafa700 1 -- 172.21.15.73:0/740619682 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8020012430 con 0x7f8028071440 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802cafa700 1 -- 172.21.15.73:0/740619682 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80200075c0 con 0x7f8028071440 2024-02-10T06:34:24.671 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 -- 172.21.15.73:0/740619682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 msgr2=0x7f80281077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 --2- 172.21.15.73:0/740619682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281077c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8020009e50 tx=0x7f8020009840 comp rx=0 tx=0).stop 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 -- 172.21.15.73:0/740619682 shutdown_connections 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 --2- 172.21.15.73:0/740619682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281077c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 -- 172.21.15.73:0/740619682 >> 172.21.15.73:0/740619682 conn(0x7f80280fcdd0 msgr2=0x7f80280ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 -- 172.21.15.73:0/740619682 shutdown_connections 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.843+0000 7f802fd60700 1 -- 172.21.15.73:0/740619682 wait complete. 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802fd60700 1 Processor -- start 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802fd60700 1 -- start start 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802fd60700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802fd60700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8028113ce0 con 0x7f8028071440 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802dafc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802dafc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281137a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57518/0 (socket says 172.21.15.73:57518) 2024-02-10T06:34:24.672 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.844+0000 7f802dafc700 1 -- 172.21.15.73:0/4001016604 learned_addr learned my addr 172.21.15.73:0/4001016604 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f802dafc700 1 -- 172.21.15.73:0/4001016604 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8020009ae0 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f802dafc700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281137a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8020009fd0 tx=0x7f8020009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80200092e0 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8020007cc0 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802001c880 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8028113ee0 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.845+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8028114b80 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.846+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8020019070 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.846+0000 7f80267fc700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f80100367e0 0x7f8010038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.846+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802805f5c0 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.846+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8020049df0 con 0x7f8028071440 2024-02-10T06:34:24.673 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.846+0000 7f802d2fb700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f80100367e0 0x7f8010038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.847+0000 7f802d2fb700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f80100367e0 0x7f8010038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f801c009990 tx=0x7f801c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:23.851+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f80200196e0 con 0x7f8028071440 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.035+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f80280ff100 con 0x7f80100367e0 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.036+0000 7f80267fc700 1 -- 172.21.15.73:0/4001016604 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f80280ff100 con 0x7f80100367e0 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f80100367e0 msgr2=0x7f8010038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f80100367e0 0x7f8010038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f801c009990 tx=0x7f801c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 msgr2=0x7f80281137a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281137a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8020009fd0 tx=0x7f8020009840 comp rx=0 tx=0).stop 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 shutdown_connections 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f80100367e0 0x7f8010038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 --2- 172.21.15.73:0/4001016604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8028071440 0x7f80281137a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 >> 172.21.15.73:0/4001016604 conn(0x7f80280fcdd0 msgr2=0x7f80280fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:24.674 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 shutdown_connections 2024-02-10T06:34:24.675 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:24.038+0000 7f802fd60700 1 -- 172.21.15.73:0/4001016604 wait complete. 2024-02-10T06:34:24.675 INFO:teuthology.orchestra.run.smithi073.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:34:24.675 INFO:teuthology.orchestra.run.smithi073.stdout:Adding key to root@localhost authorized_keys... 2024-02-10T06:34:24.675 INFO:teuthology.orchestra.run.smithi073.stdout:Adding host smithi073... 2024-02-10T06:34:24.910 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:24 smithi073 ceph-mon[29063]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:27 smithi073 ceph-mon[29063]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi073", "addr": "172.21.15.73", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:27 smithi073 ceph-mon[29063]: Deploying cephadm binary to smithi073 2024-02-10T06:34:28.086 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Added host 'smithi073' with addr '172.21.15.73' 2024-02-10T06:34:28.086 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd8077fe700 1 Processor -- start 2024-02-10T06:34:28.086 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd8077fe700 1 -- start start 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd8077fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd8001077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd8077fe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800103190 con 0x7fd800076c00 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd80559a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd8001077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd80559a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd8001077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57520/0 (socket says 172.21.15.73:57520) 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd80559a700 1 -- 172.21.15.73:0/3003665684 learned_addr learned my addr 172.21.15.73:0/3003665684 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.108+0000 7fd80559a700 1 -- 172.21.15.73:0/3003665684 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800107ce0 con 0x7fd800076c00 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd80559a700 1 --2- 172.21.15.73:0/3003665684 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd8001077a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00098e0 tx=0x7fd7f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb87a947a0da9af5 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd7f7fff700 1 -- 172.21.15.73:0/3003665684 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f000b860 con 0x7fd800076c00 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd7f7fff700 1 -- 172.21.15.73:0/3003665684 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd7f000be50 con 0x7fd800076c00 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd7f7fff700 1 -- 172.21.15.73:0/3003665684 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f0010380 con 0x7fd800076c00 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd8077fe700 1 -- 172.21.15.73:0/3003665684 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 msgr2=0x7fd8001077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd8077fe700 1 --2- 172.21.15.73:0/3003665684 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd8001077a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00098e0 tx=0x7fd7f0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:28.087 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd8077fe700 1 -- 172.21.15.73:0/3003665684 shutdown_connections 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd8077fe700 1 --2- 172.21.15.73:0/3003665684 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd8001077a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.109+0000 7fd8077fe700 1 -- 172.21.15.73:0/3003665684 >> 172.21.15.73:0/3003665684 conn(0x7fd8000fcdd0 msgr2=0x7fd8000ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd8077fe700 1 -- 172.21.15.73:0/3003665684 shutdown_connections 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd8077fe700 1 -- 172.21.15.73:0/3003665684 wait complete. 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd8077fe700 1 Processor -- start 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd8077fe700 1 -- start start 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd8077fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd800113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd8077fe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800113f90 con 0x7fd800076c00 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd80559a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd800113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd80559a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd800113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57528/0 (socket says 172.21.15.73:57528) 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.110+0000 7fd80559a700 1 -- 172.21.15.73:0/1413406949 learned_addr learned my addr 172.21.15.73:0/1413406949 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd80559a700 1 -- 172.21.15.73:0/1413406949 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00094f0 con 0x7fd800076c00 2024-02-10T06:34:28.088 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd80559a700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd800113a50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000b440 tx=0x7fd7f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f00107c0 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd7f0010db0 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f001ad30 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd800114190 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.111+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd800114e30 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.112+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd7f0010920 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.112+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80005f5c0 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.112+0000 7fd7f5ffb700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fd7ec0367e0 0x7fd7ec038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.112+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd7f004ac10 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.112+0000 7fd804d99700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fd7ec0367e0 0x7fd7ec038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.113+0000 7fd804d99700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fd7ec0367e0 0x7fd7ec038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009990 tx=0x7fd7fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.117+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd7f002d3e0 con 0x7fd800076c00 2024-02-10T06:34:28.089 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:25.304+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi073", "addr": "172.21.15.73", "target": ["mon-mgr", ""]}) v1 -- 0x7fd800108b70 con 0x7fd7ec0367e0 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.440+0000 7fd7f5ffb700 1 -- 172.21.15.73:0/1413406949 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fd800108b70 con 0x7fd7ec0367e0 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.442+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fd7ec0367e0 msgr2=0x7fd7ec038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.442+0000 7fd8077fe700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fd7ec0367e0 0x7fd7ec038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009990 tx=0x7fd7fc006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.442+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 msgr2=0x7fd800113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.442+0000 7fd8077fe700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd800113a50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000b440 tx=0x7fd7f0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.443+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 shutdown_connections 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.443+0000 7fd8077fe700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fd7ec0367e0 0x7fd7ec038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.443+0000 7fd8077fe700 1 --2- 172.21.15.73:0/1413406949 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd800076c00 0x7fd800113a50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:28.090 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.443+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 >> 172.21.15.73:0/1413406949 conn(0x7fd8000fcdd0 msgr2=0x7fd8000fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:28.091 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.443+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 shutdown_connections 2024-02-10T06:34:28.091 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:27.443+0000 7fd8077fe700 1 -- 172.21.15.73:0/1413406949 wait complete. 2024-02-10T06:34:28.091 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying mon service with default placement... 2024-02-10T06:34:28.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:28 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:28.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:28 smithi073 ceph-mon[29063]: Added host smithi073 2024-02-10T06:34:28.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:28 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:29.545 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-10T06:34:29.545 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.488+0000 7fb293107700 1 Processor -- start 2024-02-10T06:34:29.545 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.488+0000 7fb293107700 1 -- start start 2024-02-10T06:34:29.545 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.488+0000 7fb293107700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:29.545 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.488+0000 7fb293107700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1030c0 con 0x7fb28c102770 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb290ea3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb290ea3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57532/0 (socket says 172.21.15.73:57532) 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb290ea3700 1 -- 172.21.15.73:0/4190487179 learned_addr learned my addr 172.21.15.73:0/4190487179 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb290ea3700 1 -- 172.21.15.73:0/4190487179 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c103200 con 0x7fb28c102770 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb290ea3700 1 --2- 172.21.15.73:0/4190487179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c102b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb284009e50 tx=0x7fb284009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f0a8c76befe7bb server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb28b7fe700 1 -- 172.21.15.73:0/4190487179 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2840092e0 con 0x7fb28c102770 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb28b7fe700 1 -- 172.21.15.73:0/4190487179 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb284012430 con 0x7fb28c102770 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.489+0000 7fb28b7fe700 1 -- 172.21.15.73:0/4190487179 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2840075c0 con 0x7fb28c102770 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 -- 172.21.15.73:0/4190487179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 msgr2=0x7fb28c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 --2- 172.21.15.73:0/4190487179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c102b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb284009e50 tx=0x7fb284009840 comp rx=0 tx=0).stop 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 -- 172.21.15.73:0/4190487179 shutdown_connections 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 --2- 172.21.15.73:0/4190487179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c102b80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:29.546 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 -- 172.21.15.73:0/4190487179 >> 172.21.15.73:0/4190487179 conn(0x7fb28c0fded0 msgr2=0x7fb28c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 -- 172.21.15.73:0/4190487179 shutdown_connections 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.490+0000 7fb293107700 1 -- 172.21.15.73:0/4190487179 wait complete. 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb293107700 1 Processor -- start 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb293107700 1 -- start start 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb293107700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb293107700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c077190 con 0x7fb28c102770 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb290ea3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb290ea3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57540/0 (socket says 172.21.15.73:57540) 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb290ea3700 1 -- 172.21.15.73:0/1365531838 learned_addr learned my addr 172.21.15.73:0/1365531838 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb290ea3700 1 -- 172.21.15.73:0/1365531838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb284009ae0 con 0x7fb28c102770 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.491+0000 7fb290ea3700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c078d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb284000f80 tx=0x7fb284009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.492+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2840092e0 con 0x7fb28c102770 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.492+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb284007cc0 con 0x7fb28c102770 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.492+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28c077390 con 0x7fb28c102770 2024-02-10T06:34:29.547 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.492+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28401c990 con 0x7fb28c102770 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.492+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28c077880 con 0x7fb28c102770 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.492+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb284019070 con 0x7fb28c102770 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.493+0000 7fb2897fa700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fb278036790 0x7fb278038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.493+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb284049e30 con 0x7fb28c102770 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.493+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c04bed0 con 0x7fb28c102770 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.493+0000 7fb28bfff700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fb278036790 0x7fb278038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.493+0000 7fb28bfff700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fb278036790 0x7fb278038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009990 tx=0x7fb27c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.497+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb284017080 con 0x7fb28c102770 2024-02-10T06:34:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.711+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb28c078000 con 0x7fb278036790 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.718+0000 7fb2897fa700 1 -- 172.21.15.73:0/1365531838 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb28c078000 con 0x7fb278036790 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.719+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fb278036790 msgr2=0x7fb278038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.719+0000 7fb293107700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fb278036790 0x7fb278038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009990 tx=0x7fb27c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.719+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 msgr2=0x7fb28c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.719+0000 7fb293107700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c078d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb284000f80 tx=0x7fb284009840 comp rx=0 tx=0).stop 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.720+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 shutdown_connections 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.720+0000 7fb293107700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fb278036790 0x7fb278038c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.720+0000 7fb293107700 1 --2- 172.21.15.73:0/1365531838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb28c102770 0x7fb28c078d10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.720+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 >> 172.21.15.73:0/1365531838 conn(0x7fb28c0fded0 msgr2=0x7fb28c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.720+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 shutdown_connections 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:28.720+0000 7fb293107700 1 -- 172.21.15.73:0/1365531838 wait complete. 2024-02-10T06:34:29.549 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying mgr service with default placement... 2024-02-10T06:34:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:29 smithi073 ceph-mon[29063]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:29 smithi073 ceph-mon[29063]: Saving service mon spec with placement count:5 2024-02-10T06:34:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:29 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:31.232 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:31 smithi073 ceph-mon[29063]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:31.233 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:31 smithi073 ceph-mon[29063]: Saving service mgr spec with placement count:2 2024-02-10T06:34:31.233 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:31 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:31.240 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-10T06:34:31.240 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f1927a46700 1 Processor -- start 2024-02-10T06:34:31.240 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f1927a46700 1 -- start start 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f1927a46700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f1920101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f1927a46700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1920101fc0 con 0x7f1920101670 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f19257e2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f1920101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f19257e2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f1920101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41876/0 (socket says 172.21.15.73:41876) 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.986+0000 7f19257e2700 1 -- 172.21.15.73:0/4286478338 learned_addr learned my addr 172.21.15.73:0/4286478338 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f19257e2700 1 -- 172.21.15.73:0/4286478338 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1920102100 con 0x7f1920101670 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f19257e2700 1 --2- 172.21.15.73:0/4286478338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f1920101a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f191c00bb80 tx=0x7f191c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06d70dc20668977 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1917fff700 1 -- 172.21.15.73:0/4286478338 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c00dd00 con 0x7f1920101670 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1917fff700 1 -- 172.21.15.73:0/4286478338 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f191c00de60 con 0x7f1920101670 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1917fff700 1 -- 172.21.15.73:0/4286478338 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c0188a0 con 0x7f1920101670 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1927a46700 1 -- 172.21.15.73:0/4286478338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 msgr2=0x7f1920101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:31.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1927a46700 1 --2- 172.21.15.73:0/4286478338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f1920101a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f191c00bb80 tx=0x7f191c00b4f0 comp rx=0 tx=0).stop 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1927a46700 1 -- 172.21.15.73:0/4286478338 shutdown_connections 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1927a46700 1 --2- 172.21.15.73:0/4286478338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f1920101a80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.987+0000 7f1927a46700 1 -- 172.21.15.73:0/4286478338 >> 172.21.15.73:0/4286478338 conn(0x7f19200fcdd0 msgr2=0x7f19200ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f1927a46700 1 -- 172.21.15.73:0/4286478338 shutdown_connections 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f1927a46700 1 -- 172.21.15.73:0/4286478338 wait complete. 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f1927a46700 1 Processor -- start 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f1927a46700 1 -- start start 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f1927a46700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f192010f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f1927a46700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192010faf0 con 0x7f1920101670 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f19257e2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f192010f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f19257e2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f192010f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41880/0 (socket says 172.21.15.73:41880) 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.988+0000 7f19257e2700 1 -- 172.21.15.73:0/2957693333 learned_addr learned my addr 172.21.15.73:0/2957693333 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f19257e2700 1 -- 172.21.15.73:0/2957693333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c00b790 con 0x7f1920101670 2024-02-10T06:34:31.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f19257e2700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f192010f5b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f191c00d650 tx=0x7f191c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c00dc70 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f191c01d050 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192010fcf0 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f191c0223d0 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.989+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1920110990 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.990+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f191c022600 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.990+0000 7f1915ffb700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f190c0367e0 0x7f190c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.990+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f191c014070 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.990+0000 7f1924fe1700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f190c0367e0 0x7f190c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.990+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1920110180 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.991+0000 7f1924fe1700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f190c0367e0 0x7f190c038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1910009990 tx=0x7f1910006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:29.995+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f191c02c410 con 0x7f1920101670 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.202+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1920103550 con 0x7f190c0367e0 2024-02-10T06:34:31.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.209+0000 7f1915ffb700 1 -- 172.21.15.73:0/2957693333 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1920103550 con 0x7f190c0367e0 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f190c0367e0 msgr2=0x7f190c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f190c0367e0 0x7f190c038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1910009990 tx=0x7f1910006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 msgr2=0x7f192010f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f192010f5b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f191c00d650 tx=0x7f191c00b4f0 comp rx=0 tx=0).stop 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 shutdown_connections 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f190c0367e0 0x7f190c038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 --2- 172.21.15.73:0/2957693333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1920101670 0x7f192010f5b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.211+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 >> 172.21.15.73:0/2957693333 conn(0x7f19200fcdd0 msgr2=0x7f19201063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.212+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 shutdown_connections 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:30.212+0000 7f1927a46700 1 -- 172.21.15.73:0/2957693333 wait complete. 2024-02-10T06:34:31.244 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying crash service with default placement... 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748ea9a700 1 Processor -- start 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748ea9a700 1 -- start start 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748ea9a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488104230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748ea9a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488104770 con 0x7f7488103e20 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748c836700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748c836700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41892/0 (socket says 172.21.15.73:41892) 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.649+0000 7f748c836700 1 -- 172.21.15.73:0/1312816177 learned_addr learned my addr 172.21.15.73:0/1312816177 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748c836700 1 -- 172.21.15.73:0/1312816177 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74881048b0 con 0x7f7488103e20 2024-02-10T06:34:32.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748c836700 1 --2- 172.21.15.73:0/1312816177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488104230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f747c0098e0 tx=0x7f747c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de6f47df3e0df66 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f74877fe700 1 -- 172.21.15.73:0/1312816177 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f747c00b860 con 0x7f7488103e20 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f74877fe700 1 -- 172.21.15.73:0/1312816177 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f747c00be50 con 0x7f7488103e20 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f74877fe700 1 -- 172.21.15.73:0/1312816177 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f747c010380 con 0x7f7488103e20 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748ea9a700 1 -- 172.21.15.73:0/1312816177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 msgr2=0x7f7488104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748ea9a700 1 --2- 172.21.15.73:0/1312816177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488104230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f747c0098e0 tx=0x7f747c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748ea9a700 1 -- 172.21.15.73:0/1312816177 shutdown_connections 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748ea9a700 1 --2- 172.21.15.73:0/1312816177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488104230 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.650+0000 7f748ea9a700 1 -- 172.21.15.73:0/1312816177 >> 172.21.15.73:0/1312816177 conn(0x7f74880fc580 msgr2=0x7f74880fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748ea9a700 1 -- 172.21.15.73:0/1312816177 shutdown_connections 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748ea9a700 1 -- 172.21.15.73:0/1312816177 wait complete. 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748ea9a700 1 Processor -- start 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748ea9a700 1 -- start start 2024-02-10T06:34:32.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748ea9a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488111800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748ea9a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488111d40 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748c836700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488111800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748c836700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488111800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41906/0 (socket says 172.21.15.73:41906) 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.651+0000 7f748c836700 1 -- 172.21.15.73:0/1249980614 learned_addr learned my addr 172.21.15.73:0/1249980614 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f748c836700 1 -- 172.21.15.73:0/1249980614 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0094f0 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f748c836700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488111800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b440 tx=0x7f747c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f747c0107c0 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f747c010db0 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7488111f40 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f747c01ada0 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.652+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7488112be0 con 0x7f7488103e20 2024-02-10T06:34:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.653+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f747c02e440 con 0x7f7488103e20 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.653+0000 7f74857fa700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f74780367e0 0x7f7478038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.653+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748805f5c0 con 0x7f7488103e20 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.653+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f747c04ade0 con 0x7f7488103e20 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.653+0000 7f7487fff700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f74780367e0 0x7f7478038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.654+0000 7f7487fff700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f74780367e0 0x7f7478038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7474009990 tx=0x7f7474006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.658+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f747c02c570 con 0x7f7488103e20 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.853+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f7488101440 con 0x7f74780367e0 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.857+0000 7f74857fa700 1 -- 172.21.15.73:0/1249980614 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f7488101440 con 0x7f74780367e0 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f74780367e0 msgr2=0x7f7478038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:32.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f74780367e0 0x7f7478038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7474009990 tx=0x7f7474006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 msgr2=0x7f7488111800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488111800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b440 tx=0x7f747c009250 comp rx=0 tx=0).stop 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 shutdown_connections 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f74780367e0 0x7f7478038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 --2- 172.21.15.73:0/1249980614 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7488103e20 0x7f7488111800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 >> 172.21.15.73:0/1249980614 conn(0x7f74880fc580 msgr2=0x7f74880fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 shutdown_connections 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:31.859+0000 7f748ea9a700 1 -- 172.21.15.73:0/1249980614 wait complete. 2024-02-10T06:34:32.784 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying prometheus service with default placement... 2024-02-10T06:34:33.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:32 smithi073 ceph-mon[29063]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:33.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:32 smithi073 ceph-mon[29063]: Saving service crash spec with placement * 2024-02-10T06:34:33.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:32 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:33.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:32 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.166+0000 7f44ffa79700 1 Processor -- start 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44ffa79700 1 -- start start 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44ffa79700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44ffa79700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f81030c0 con 0x7f44f8102770 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44fd815700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44fd815700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41912/0 (socket says 172.21.15.73:41912) 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44fd815700 1 -- 172.21.15.73:0/830389848 learned_addr learned my addr 172.21.15.73:0/830389848 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:33.991 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44fd815700 1 -- 172.21.15.73:0/830389848 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8103200 con 0x7f44f8102770 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.167+0000 7f44fd815700 1 --2- 172.21.15.73:0/830389848 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f8102b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f44e80098e0 tx=0x7f44e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ec4b1742bbc4e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44fc813700 1 -- 172.21.15.73:0/830389848 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e800b860 con 0x7f44f8102770 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44fc813700 1 -- 172.21.15.73:0/830389848 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44e800be90 con 0x7f44f8102770 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44fc813700 1 -- 172.21.15.73:0/830389848 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e80103c0 con 0x7f44f8102770 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 -- 172.21.15.73:0/830389848 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 msgr2=0x7f44f8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 --2- 172.21.15.73:0/830389848 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f8102b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f44e80098e0 tx=0x7f44e8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 -- 172.21.15.73:0/830389848 shutdown_connections 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 --2- 172.21.15.73:0/830389848 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f8102b80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 -- 172.21.15.73:0/830389848 >> 172.21.15.73:0/830389848 conn(0x7f44f80fded0 msgr2=0x7f44f8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 -- 172.21.15.73:0/830389848 shutdown_connections 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.168+0000 7f44ffa79700 1 -- 172.21.15.73:0/830389848 wait complete. 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44ffa79700 1 Processor -- start 2024-02-10T06:34:33.992 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44ffa79700 1 -- start start 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44ffa79700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f81115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44ffa79700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8111af0 con 0x7f44f8102770 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44fd815700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f81115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44fd815700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f81115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41928/0 (socket says 172.21.15.73:41928) 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44fd815700 1 -- 172.21.15.73:0/240287819 learned_addr learned my addr 172.21.15.73:0/240287819 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44fd815700 1 -- 172.21.15.73:0/240287819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80094f0 con 0x7f44f8102770 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.169+0000 7f44fd815700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f81115b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f44e80090b0 tx=0x7f44e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.170+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e8010840 con 0x7f44f8102770 2024-02-10T06:34:33.995 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.170+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44e8010e70 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.170+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e8024e40 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.170+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f8111cf0 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.170+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f8112990 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.170+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f805f5c0 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.171+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f44e8024430 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.171+0000 7f44ee7fc700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f44e4036790 0x7f44e4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.171+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f44e804a750 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.171+0000 7f44fd014700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f44e4036790 0x7f44e4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.171+0000 7f44fd014700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f44e4036790 0x7f44e4038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009990 tx=0x7f44f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.175+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f44e802c930 con 0x7f44f8102770 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.380+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f44f81001b0 con 0x7f44e4036790 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.385+0000 7f44ee7fc700 1 -- 172.21.15.73:0/240287819 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f44f81001b0 con 0x7f44e4036790 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f44e4036790 msgr2=0x7f44e4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:33.996 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f44e4036790 0x7f44e4038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009990 tx=0x7f44f4006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 msgr2=0x7f44f81115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f81115b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f44e80090b0 tx=0x7f44e8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 shutdown_connections 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f44e4036790 0x7f44e4038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 --2- 172.21.15.73:0/240287819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f44f8102770 0x7f44f81115b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 >> 172.21.15.73:0/240287819 conn(0x7f44f80fded0 msgr2=0x7f44f80ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 shutdown_connections 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:33.387+0000 7f44ffa79700 1 -- 172.21.15.73:0/240287819 wait complete. 2024-02-10T06:34:33.997 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying grafana service with default placement... 2024-02-10T06:34:34.616 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:34 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:34.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:34 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:35.434 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:35 smithi073 ceph-mon[29063]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:35.434 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:35 smithi073 ceph-mon[29063]: Saving service prometheus spec with placement count:1 2024-02-10T06:34:35.434 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:35 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f37636fe700 1 Processor -- start 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f37636fe700 1 -- start start 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f37636fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f37636fe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c101fc0 con 0x7f375c101670 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f376149a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:35.442 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f376149a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41936/0 (socket says 172.21.15.73:41936) 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.399+0000 7f376149a700 1 -- 172.21.15.73:0/1277396763 learned_addr learned my addr 172.21.15.73:0/1277396763 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.400+0000 7f376149a700 1 -- 172.21.15.73:0/1277396763 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c102100 con 0x7f375c101670 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.400+0000 7f376149a700 1 --2- 172.21.15.73:0/1277396763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c101a80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3758009e50 tx=0x7f3758009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516cf4ea4ad42568 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.400+0000 7f3753fff700 1 -- 172.21.15.73:0/1277396763 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37580092e0 con 0x7f375c101670 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.400+0000 7f3753fff700 1 -- 172.21.15.73:0/1277396763 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3758012430 con 0x7f375c101670 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.400+0000 7f3753fff700 1 -- 172.21.15.73:0/1277396763 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37580075a0 con 0x7f375c101670 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- 172.21.15.73:0/1277396763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 msgr2=0x7f375c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 --2- 172.21.15.73:0/1277396763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c101a80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3758009e50 tx=0x7f3758009840 comp rx=0 tx=0).stop 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- 172.21.15.73:0/1277396763 shutdown_connections 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 --2- 172.21.15.73:0/1277396763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c101a80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- 172.21.15.73:0/1277396763 >> 172.21.15.73:0/1277396763 conn(0x7f375c0fcdd0 msgr2=0x7f375c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- 172.21.15.73:0/1277396763 shutdown_connections 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- 172.21.15.73:0/1277396763 wait complete. 2024-02-10T06:34:35.443 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 Processor -- start 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- start start 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c114560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.401+0000 7f37636fe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c076c50 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f376149a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c114560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f376149a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c114560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41948/0 (socket says 172.21.15.73:41948) 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f376149a700 1 -- 172.21.15.73:0/1883471153 learned_addr learned my addr 172.21.15.73:0/1883471153 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f376149a700 1 -- 172.21.15.73:0/1883471153 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3758009ae0 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f376149a700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c114560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3758012800 tx=0x7f3758009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37580092e0 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3758007d30 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c076e50 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37580259a0 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.402+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c114ce0 con 0x7f375c101670 2024-02-10T06:34:35.444 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.403+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3758019050 con 0x7f375c101670 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.403+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c10a590 con 0x7f375c101670 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.403+0000 7f3751ffb700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f37480367e0 0x7f3748038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.403+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f375804a400 con 0x7f375c101670 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.403+0000 7f3760c99700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f37480367e0 0x7f3748038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.404+0000 7f3760c99700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f37480367e0 0x7f3748038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f374c009990 tx=0x7f374c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.408+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f375802dba0 con 0x7f375c101670 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.615+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f375c05e3d0 con 0x7f37480367e0 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.620+0000 7f3751ffb700 1 -- 172.21.15.73:0/1883471153 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f375c05e3d0 con 0x7f37480367e0 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f37480367e0 msgr2=0x7f3748038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f37480367e0 0x7f3748038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f374c009990 tx=0x7f374c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 msgr2=0x7f375c114560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c114560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3758012800 tx=0x7f3758009840 comp rx=0 tx=0).stop 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 shutdown_connections 2024-02-10T06:34:35.445 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f37480367e0 0x7f3748038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:35.446 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 --2- 172.21.15.73:0/1883471153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f375c101670 0x7f375c114560 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:35.446 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 >> 172.21.15.73:0/1883471153 conn(0x7f375c0fcdd0 msgr2=0x7f375c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:35.446 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 shutdown_connections 2024-02-10T06:34:35.446 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:34.622+0000 7f37636fe700 1 -- 172.21.15.73:0/1883471153 wait complete. 2024-02-10T06:34:35.446 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying node-exporter service with default placement... 2024-02-10T06:34:36.364 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:36 smithi073 ceph-mon[29063]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:36.364 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:36 smithi073 ceph-mon[29063]: Saving service grafana spec with placement count:1 2024-02-10T06:34:36.365 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:36 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:37.018 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8686f98700 1 Processor -- start 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8686f98700 1 -- start start 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8686f98700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f8680102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8686f98700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86801030c0 con 0x7f8680102770 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8684d34700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f8680102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8684d34700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f8680102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41954/0 (socket says 172.21.15.73:41954) 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8684d34700 1 -- 172.21.15.73:0/1414453080 learned_addr learned my addr 172.21.15.73:0/1414453080 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.149+0000 7f8684d34700 1 -- 172.21.15.73:0/1414453080 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680103200 con 0x7f8680102770 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8684d34700 1 --2- 172.21.15.73:0/1414453080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f8680102b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86740098e0 tx=0x7f8674009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed032cf422ef9fc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f867f7fe700 1 -- 172.21.15.73:0/1414453080 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f867400b860 con 0x7f8680102770 2024-02-10T06:34:37.023 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f867f7fe700 1 -- 172.21.15.73:0/1414453080 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f867400be90 con 0x7f8680102770 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f867f7fe700 1 -- 172.21.15.73:0/1414453080 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86740103c0 con 0x7f8680102770 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 -- 172.21.15.73:0/1414453080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 msgr2=0x7f8680102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 --2- 172.21.15.73:0/1414453080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f8680102b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86740098e0 tx=0x7f8674009250 comp rx=0 tx=0).stop 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 -- 172.21.15.73:0/1414453080 shutdown_connections 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 --2- 172.21.15.73:0/1414453080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f8680102b80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 -- 172.21.15.73:0/1414453080 >> 172.21.15.73:0/1414453080 conn(0x7f86800fded0 msgr2=0x7f8680100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:37.024 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 -- 172.21.15.73:0/1414453080 shutdown_connections 2024-02-10T06:34:37.025 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.150+0000 7f8686f98700 1 -- 172.21.15.73:0/1414453080 wait complete. 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8686f98700 1 Processor -- start 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8686f98700 1 -- start start 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8686f98700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f86801115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8686f98700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8680111af0 con 0x7f8680102770 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8684d34700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f86801115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8684d34700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f86801115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41970/0 (socket says 172.21.15.73:41970) 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.151+0000 7f8684d34700 1 -- 172.21.15.73:0/3623028019 learned_addr learned my addr 172.21.15.73:0/3623028019 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:37.026 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f8684d34700 1 -- 172.21.15.73:0/3623028019 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86740094f0 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f8684d34700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f86801115b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8674011680 tx=0x7f8674009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8674010890 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8680111cf0 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8674010ec0 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86740243f0 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.152+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8680112990 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.153+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f867402d3f0 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.153+0000 7f867d7fa700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f8670036790 0x7f8670038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.153+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f867404a270 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.153+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868005f5c0 con 0x7f8680102770 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.153+0000 7f867ffff700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f8670036790 0x7f8670038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.153+0000 7f867ffff700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f8670036790 0x7f8670038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f866c009990 tx=0x7f866c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:37.027 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.157+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f867402b8e0 con 0x7f8680102770 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.363+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f86801001b0 con 0x7f8670036790 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.378+0000 7f867d7fa700 1 -- 172.21.15.73:0/3623028019 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f86801001b0 con 0x7f8670036790 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f8670036790 msgr2=0x7f8670038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f8670036790 0x7f8670038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f866c009990 tx=0x7f866c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 msgr2=0x7f86801115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f86801115b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8674011680 tx=0x7f8674009250 comp rx=0 tx=0).stop 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 shutdown_connections 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f8670036790 0x7f8670038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 --2- 172.21.15.73:0/3623028019 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8680102770 0x7f86801115b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 >> 172.21.15.73:0/3623028019 conn(0x7f86800fded0 msgr2=0x7f86800ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 shutdown_connections 2024-02-10T06:34:37.028 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:36.381+0000 7f8686f98700 1 -- 172.21.15.73:0/3623028019 wait complete. 2024-02-10T06:34:37.029 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying alertmanager service with default placement... 2024-02-10T06:34:37.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:37 smithi073 ceph-mon[29063]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:37.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:37 smithi073 ceph-mon[29063]: Saving service node-exporter spec with placement * 2024-02-10T06:34:37.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:37 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effccf41700 1 Processor -- start 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effccf41700 1 -- start start 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effccf41700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effccf41700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc81030c0 con 0x7effc8102770 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effc659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effc659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41986/0 (socket says 172.21.15.73:41986) 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effc659c700 1 -- 172.21.15.73:0/2551914815 learned_addr learned my addr 172.21.15.73:0/2551914815 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.442+0000 7effc659c700 1 -- 172.21.15.73:0/2551914815 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8103200 con 0x7effc8102770 2024-02-10T06:34:38.266 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effc659c700 1 --2- 172.21.15.73:0/2551914815 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc8102b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7effb80098e0 tx=0x7effb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef65465206aada server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effc559a700 1 -- 172.21.15.73:0/2551914815 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb800b860 con 0x7effc8102770 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effc559a700 1 -- 172.21.15.73:0/2551914815 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effb800be90 con 0x7effc8102770 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effc559a700 1 -- 172.21.15.73:0/2551914815 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb80103c0 con 0x7effc8102770 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effccf41700 1 -- 172.21.15.73:0/2551914815 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 msgr2=0x7effc8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effccf41700 1 --2- 172.21.15.73:0/2551914815 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc8102b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7effb80098e0 tx=0x7effb8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effccf41700 1 -- 172.21.15.73:0/2551914815 shutdown_connections 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effccf41700 1 --2- 172.21.15.73:0/2551914815 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc8102b80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effccf41700 1 -- 172.21.15.73:0/2551914815 >> 172.21.15.73:0/2551914815 conn(0x7effc80fded0 msgr2=0x7effc8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.443+0000 7effccf41700 1 -- 172.21.15.73:0/2551914815 shutdown_connections 2024-02-10T06:34:38.267 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effccf41700 1 -- 172.21.15.73:0/2551914815 wait complete. 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effccf41700 1 Processor -- start 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effccf41700 1 -- start start 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effccf41700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc81115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effccf41700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc8111af0 con 0x7effc8102770 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effc659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc81115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effc659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc81115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41992/0 (socket says 172.21.15.73:41992) 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effc659c700 1 -- 172.21.15.73:0/1054721492 learned_addr learned my addr 172.21.15.73:0/1054721492 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.444+0000 7effc659c700 1 -- 172.21.15.73:0/1054721492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80094f0 con 0x7effc8102770 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effc659c700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc81115b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7effb80090b0 tx=0x7effb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb8010840 con 0x7effc8102770 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc8111cf0 con 0x7effc8102770 2024-02-10T06:34:38.268 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effb8010e70 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb8024e40 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc8112990 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.445+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7effb801a510 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.446+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc805f5c0 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.446+0000 7effb6ffd700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7effb0036740 0x7effb0038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.446+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7effb804a7d0 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.446+0000 7effc5d9b700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7effb0036740 0x7effb0038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.446+0000 7effc5d9b700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7effb0036740 0x7effb0038bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7effbc009990 tx=0x7effbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.450+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7effb802c930 con 0x7effc8102770 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.662+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7effc81001b0 con 0x7effb0036740 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.669+0000 7effb6ffd700 1 -- 172.21.15.73:0/1054721492 <== mgr.14120 v2:172.21.15.73:6800/1157895579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7effc81001b0 con 0x7effb0036740 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7effb0036740 msgr2=0x7effb0038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7effb0036740 0x7effb0038bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7effbc009990 tx=0x7effbc006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:38.269 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 msgr2=0x7effc81115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc81115b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7effb80090b0 tx=0x7effb8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 shutdown_connections 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7effb0036740 0x7effb0038bf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 --2- 172.21.15.73:0/1054721492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effc8102770 0x7effc81115b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 >> 172.21.15.73:0/1054721492 conn(0x7effc80fded0 msgr2=0x7effc80ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 shutdown_connections 2024-02-10T06:34:38.270 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:37.671+0000 7effccf41700 1 -- 172.21.15.73:0/1054721492 wait complete. 2024-02-10T06:34:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:38 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:38 smithi073 ceph-mon[29063]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:38 smithi073 ceph-mon[29063]: Saving service alertmanager spec with placement count:1 2024-02-10T06:34:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:38 smithi073 ceph-mon[29063]: from='mgr.14120 172.21.15.73:0/1636712088' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:38 smithi073 ceph-mon[29063]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:34:39.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.700+0000 7f3c85e23700 1 Processor -- start 2024-02-10T06:34:39.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.700+0000 7f3c85e23700 1 -- start start 2024-02-10T06:34:39.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.700+0000 7f3c85e23700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c80077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:39.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.700+0000 7f3c85e23700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80079110 con 0x7f3c80078d00 2024-02-10T06:34:39.650 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c80077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c80077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46550/0 (socket says 172.21.15.73:46550) 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7f7fe700 1 -- 172.21.15.73:0/1193767428 learned_addr learned my addr 172.21.15.73:0/1193767428 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7f7fe700 1 -- 172.21.15.73:0/1193767428 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80077750 con 0x7f3c80078d00 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7f7fe700 1 --2- 172.21.15.73:0/1193767428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c80077210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3c700098e0 tx=0x7f3c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cff683e1453779 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7effd700 1 -- 172.21.15.73:0/1193767428 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7000b860 con 0x7f3c80078d00 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7effd700 1 -- 172.21.15.73:0/1193767428 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c7000be90 con 0x7f3c80078d00 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.701+0000 7f3c7effd700 1 -- 172.21.15.73:0/1193767428 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c700103c0 con 0x7f3c80078d00 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 -- 172.21.15.73:0/1193767428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 msgr2=0x7f3c80077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 --2- 172.21.15.73:0/1193767428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c80077210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3c700098e0 tx=0x7f3c70009250 comp rx=0 tx=0).stop 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 -- 172.21.15.73:0/1193767428 shutdown_connections 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 --2- 172.21.15.73:0/1193767428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c80077210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 -- 172.21.15.73:0/1193767428 >> 172.21.15.73:0/1193767428 conn(0x7f3c800fded0 msgr2=0x7f3c80100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 -- 172.21.15.73:0/1193767428 shutdown_connections 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 -- 172.21.15.73:0/1193767428 wait complete. 2024-02-10T06:34:39.651 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.702+0000 7f3c85e23700 1 Processor -- start 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c85e23700 1 -- start start 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c85e23700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c801137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c85e23700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80113d30 con 0x7f3c80078d00 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c801137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c801137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46560/0 (socket says 172.21.15.73:46560) 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7f7fe700 1 -- 172.21.15.73:0/3229435118 learned_addr learned my addr 172.21.15.73:0/3229435118 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7f7fe700 1 -- 172.21.15.73:0/3229435118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700094f0 con 0x7f3c80078d00 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7f7fe700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c801137f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c70006d40 tx=0x7f3c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c70010840 con 0x7f3c80078d00 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c70010e70 con 0x7f3c80078d00 2024-02-10T06:34:39.652 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c80113f30 con 0x7f3c80078d00 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c70024ea0 con 0x7f3c80078d00 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.703+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c80114bd0 con 0x7f3c80078d00 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.704+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3c7001a4c0 con 0x7f3c80078d00 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.704+0000 7f3c7cff9700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f3c6c036790 0x7f3c6c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.704+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3c7004a700 con 0x7f3c80078d00 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.704+0000 7f3c77fff700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f3c6c036790 0x7f3c6c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.704+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c801143c0 con 0x7f3c80078d00 2024-02-10T06:34:39.653 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.705+0000 7f3c77fff700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f3c6c036790 0x7f3c6c038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009990 tx=0x7f3c68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.709+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3c7002e440 con 0x7f3c80078d00 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.906+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f3c8005f5c0 con 0x7f3c80078d00 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.912+0000 7f3c7cff9700 1 -- 172.21.15.73:0/3229435118 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3c70048020 con 0x7f3c80078d00 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f3c6c036790 msgr2=0x7f3c6c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f3c6c036790 0x7f3c6c038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009990 tx=0x7f3c68006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 msgr2=0x7f3c801137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c801137f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c70006d40 tx=0x7f3c70009250 comp rx=0 tx=0).stop 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 shutdown_connections 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f3c6c036790 0x7f3c6c038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 --2- 172.21.15.73:0/3229435118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c80078d00 0x7f3c801137f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 >> 172.21.15.73:0/3229435118 conn(0x7f3c800fded0 msgr2=0x7f3c800ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 shutdown_connections 2024-02-10T06:34:39.654 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:38.914+0000 7f3c85e23700 1 -- 172.21.15.73:0/3229435118 wait complete. 2024-02-10T06:34:40.171 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:39 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3229435118' entity='client.admin' 2024-02-10T06:34:41.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:41 smithi073 ceph-mon[29063]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:34:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:41 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2473577037' entity='client.admin' 2024-02-10T06:34:41.514 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.071+0000 7fefbcbec700 1 Processor -- start 2024-02-10T06:34:41.514 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.071+0000 7fefbcbec700 1 -- start start 2024-02-10T06:34:41.514 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.071+0000 7fefbcbec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:41.515 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.071+0000 7fefbcbec700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81030c0 con 0x7fefb8102770 2024-02-10T06:34:41.515 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:41.515 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46562/0 (socket says 172.21.15.73:46562) 2024-02-10T06:34:41.515 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb659c700 1 -- 172.21.15.73:0/1178900335 learned_addr learned my addr 172.21.15.73:0/1178900335 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:41.515 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb659c700 1 -- 172.21.15.73:0/1178900335 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8103200 con 0x7fefb8102770 2024-02-10T06:34:41.515 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb659c700 1 --2- 172.21.15.73:0/1178900335 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8102b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fefac0098e0 tx=0x7fefac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449f4497d624a747 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:41.516 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb559a700 1 -- 172.21.15.73:0/1178900335 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac00b860 con 0x7fefb8102770 2024-02-10T06:34:41.516 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb559a700 1 -- 172.21.15.73:0/1178900335 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefac00be90 con 0x7fefb8102770 2024-02-10T06:34:41.516 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.072+0000 7fefb559a700 1 -- 172.21.15.73:0/1178900335 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac0103c0 con 0x7fefb8102770 2024-02-10T06:34:41.516 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 -- 172.21.15.73:0/1178900335 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 msgr2=0x7fefb8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:41.516 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 --2- 172.21.15.73:0/1178900335 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8102b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fefac0098e0 tx=0x7fefac009250 comp rx=0 tx=0).stop 2024-02-10T06:34:41.516 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 -- 172.21.15.73:0/1178900335 shutdown_connections 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 --2- 172.21.15.73:0/1178900335 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8102b80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 -- 172.21.15.73:0/1178900335 >> 172.21.15.73:0/1178900335 conn(0x7fefb80fded0 msgr2=0x7fefb8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 -- 172.21.15.73:0/1178900335 shutdown_connections 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 -- 172.21.15.73:0/1178900335 wait complete. 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 Processor -- start 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.073+0000 7fefbcbec700 1 -- start start 2024-02-10T06:34:41.517 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefbcbec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:41.518 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefbcbec700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8076080 con 0x7fefb8102770 2024-02-10T06:34:41.518 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefb659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:41.518 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefb659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46568/0 (socket says 172.21.15.73:46568) 2024-02-10T06:34:41.518 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefb659c700 1 -- 172.21.15.73:0/2473577037 learned_addr learned my addr 172.21.15.73:0/2473577037 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:41.518 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefb659c700 1 -- 172.21.15.73:0/2473577037 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac0094f0 con 0x7fefb8102770 2024-02-10T06:34:41.518 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefb659c700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8077c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefac006b00 tx=0x7fefac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:41.519 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac0109a0 con 0x7fefb8102770 2024-02-10T06:34:41.519 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb8076280 con 0x7fefb8102770 2024-02-10T06:34:41.519 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefac01a430 con 0x7fefb8102770 2024-02-10T06:34:41.519 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac0245b0 con 0x7fefb8102770 2024-02-10T06:34:41.519 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.074+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb8076710 con 0x7fefb8102770 2024-02-10T06:34:41.520 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.075+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fefac024830 con 0x7fefb8102770 2024-02-10T06:34:41.520 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.075+0000 7fefa6ffd700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fefa0036790 0x7fefa0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:41.520 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.075+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fefac04a6d0 con 0x7fefb8102770 2024-02-10T06:34:41.520 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.075+0000 7fefb5d9b700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fefa0036790 0x7fefa0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:41.520 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.075+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb810b690 con 0x7fefb8102770 2024-02-10T06:34:41.521 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.076+0000 7fefb5d9b700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fefa0036790 0x7fefa0038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009990 tx=0x7fefa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:41.521 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.080+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fefac018db0 con 0x7fefb8102770 2024-02-10T06:34:41.521 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.268+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fefb805f5c0 con 0x7fefb8102770 2024-02-10T06:34:41.521 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.272+0000 7fefa6ffd700 1 -- 172.21.15.73:0/2473577037 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fefac018350 con 0x7fefb8102770 2024-02-10T06:34:41.521 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fefa0036790 msgr2=0x7fefa0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:41.521 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fefa0036790 0x7fefa0038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009990 tx=0x7fefa8006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:41.522 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 msgr2=0x7fefb8077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:41.522 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8077c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefac006b00 tx=0x7fefac009250 comp rx=0 tx=0).stop 2024-02-10T06:34:41.522 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 shutdown_connections 2024-02-10T06:34:41.522 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fefa0036790 0x7fefa0038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:41.522 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 --2- 172.21.15.73:0/2473577037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fefb8102770 0x7fefb8077c00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:41.522 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 >> 172.21.15.73:0/2473577037 conn(0x7fefb80fded0 msgr2=0x7fefb80fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:41.523 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.274+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 shutdown_connections 2024-02-10T06:34:41.523 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:40.275+0000 7fefbcbec700 1 -- 172.21.15.73:0/2473577037 wait complete. 2024-02-10T06:34:41.523 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling the dashboard module... 2024-02-10T06:34:42.578 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:42 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2865320364' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T06:34:42.942 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.959+0000 7f5c910f5700 1 Processor -- start 2024-02-10T06:34:42.942 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.959+0000 7f5c910f5700 1 -- start start 2024-02-10T06:34:42.942 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c910f5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c1031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c910f5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c103700 con 0x7f5c8c102db0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c8ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c1031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c8ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c1031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46570/0 (socket says 172.21.15.73:46570) 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c8ad9d700 1 -- 172.21.15.73:0/1985737655 learned_addr learned my addr 172.21.15.73:0/1985737655 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c8ad9d700 1 -- 172.21.15.73:0/1985737655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c103840 con 0x7f5c8c102db0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c8ad9d700 1 --2- 172.21.15.73:0/1985737655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c1031c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c800098e0 tx=0x7f5c80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd75846366b9feca server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c89d9b700 1 -- 172.21.15.73:0/1985737655 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c8000b860 con 0x7f5c8c102db0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c89d9b700 1 -- 172.21.15.73:0/1985737655 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c8000be90 con 0x7f5c8c102db0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.960+0000 7f5c89d9b700 1 -- 172.21.15.73:0/1985737655 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c800103c0 con 0x7f5c8c102db0 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 -- 172.21.15.73:0/1985737655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 msgr2=0x7f5c8c1031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 --2- 172.21.15.73:0/1985737655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c1031c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c800098e0 tx=0x7f5c80009250 comp rx=0 tx=0).stop 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 -- 172.21.15.73:0/1985737655 shutdown_connections 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 --2- 172.21.15.73:0/1985737655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c1031c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:42.943 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 -- 172.21.15.73:0/1985737655 >> 172.21.15.73:0/1985737655 conn(0x7f5c8c0fc580 msgr2=0x7f5c8c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 -- 172.21.15.73:0/1985737655 shutdown_connections 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.961+0000 7f5c910f5700 1 -- 172.21.15.73:0/1985737655 wait complete. 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c910f5700 1 Processor -- start 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c910f5700 1 -- start start 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c910f5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c910f5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c113f90 con 0x7f5c8c102db0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c8ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c8ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46580/0 (socket says 172.21.15.73:46580) 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c8ad9d700 1 -- 172.21.15.73:0/2865320364 learned_addr learned my addr 172.21.15.73:0/2865320364 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c8ad9d700 1 -- 172.21.15.73:0/2865320364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c800094f0 con 0x7f5c8c102db0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c8ad9d700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c113a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006b00 tx=0x7f5c80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c800109a0 con 0x7f5c8c102db0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c8001a430 con 0x7f5c8c102db0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c114190 con 0x7f5c8c102db0 2024-02-10T06:34:42.944 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c800245b0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.962+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c114e30 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.963+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5c80024830 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.963+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c05f5c0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.963+0000 7f5c737fe700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f5c74036790 0x7f5c74038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.963+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5c8004a7e0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.964+0000 7f5c8a59c700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f5c74036790 0x7f5c74038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.964+0000 7f5c8a59c700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f5c74036790 0x7f5c74038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009990 tx=0x7f5c7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:41.968+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5c800189a0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.202+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f5c8c04bed0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.275+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f5c8001aaa0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.277+0000 7f5c737fe700 1 -- 172.21.15.73:0/2865320364 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f5c8002d9f0 con 0x7f5c8c102db0 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f5c74036790 msgr2=0x7f5c74038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f5c74036790 0x7f5c74038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009990 tx=0x7f5c7c006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:42.945 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 msgr2=0x7f5c8c113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c113a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006b00 tx=0x7f5c80009250 comp rx=0 tx=0).stop 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 shutdown_connections 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7f5c74036790 0x7f5c74038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 --2- 172.21.15.73:0/2865320364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c8c102db0 0x7f5c8c113a50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 >> 172.21.15.73:0/2865320364 conn(0x7f5c8c0fc580 msgr2=0x7f5c8c0fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 shutdown_connections 2024-02-10T06:34:42.946 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:42.279+0000 7f5c910f5700 1 -- 172.21.15.73:0/2865320364 wait complete. 2024-02-10T06:34:43.414 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:43 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2865320364' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T06:34:43.414 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:43 smithi073 ceph-mon[29063]: mgrmap e9: smithi073.vjivfn(active, since 24s) 2024-02-10T06:34:44.328 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "active_name": "smithi073.vjivfn", 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.413+0000 7fcb49fa7700 1 Processor -- start 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.413+0000 7fcb49fa7700 1 -- start start 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.413+0000 7fcb49fa7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44104190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.413+0000 7fcb49fa7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441046d0 con 0x7fcb44103d80 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb437fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:44.329 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb437fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46602/0 (socket says 172.21.15.73:46602) 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb437fe700 1 -- 172.21.15.73:0/3729369278 learned_addr learned my addr 172.21.15.73:0/3729369278 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb437fe700 1 -- 172.21.15.73:0/3729369278 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44104810 con 0x7fcb44103d80 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb437fe700 1 --2- 172.21.15.73:0/3729369278 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44104190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcb340098e0 tx=0x7fcb34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4320339461b4f03b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb427fc700 1 -- 172.21.15.73:0/3729369278 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3400b860 con 0x7fcb44103d80 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb427fc700 1 -- 172.21.15.73:0/3729369278 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb3400be90 con 0x7fcb44103d80 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.414+0000 7fcb427fc700 1 -- 172.21.15.73:0/3729369278 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb340103c0 con 0x7fcb44103d80 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 -- 172.21.15.73:0/3729369278 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 msgr2=0x7fcb44104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 --2- 172.21.15.73:0/3729369278 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44104190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcb340098e0 tx=0x7fcb34009250 comp rx=0 tx=0).stop 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 -- 172.21.15.73:0/3729369278 shutdown_connections 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 --2- 172.21.15.73:0/3729369278 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44104190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 -- 172.21.15.73:0/3729369278 >> 172.21.15.73:0/3729369278 conn(0x7fcb440fc580 msgr2=0x7fcb440fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 -- 172.21.15.73:0/3729369278 shutdown_connections 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 -- 172.21.15.73:0/3729369278 wait complete. 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.415+0000 7fcb49fa7700 1 Processor -- start 2024-02-10T06:34:44.330 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb49fa7700 1 -- start start 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb49fa7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb49fa7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44111db0 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb437fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb437fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44111870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46616/0 (socket says 172.21.15.73:46616) 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb437fe700 1 -- 172.21.15.73:0/574963442 learned_addr learned my addr 172.21.15.73:0/574963442 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb437fe700 1 -- 172.21.15.73:0/574963442 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340094f0 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb437fe700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44111870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb34011650 tx=0x7fcb34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb34010890 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb34010ec0 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb44111fb0 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3401a430 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.416+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb44112c50 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.417+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fcb3401a6b0 con 0x7fcb44103d80 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.417+0000 7fcb2bfff700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 0x7fcb2c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:44.331 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.417+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcb3404b5f0 con 0x7fcb44103d80 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.417+0000 7fcb42ffd700 1 -- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 msgr2=0x7fcb2c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.417+0000 7fcb42ffd700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 0x7fcb2c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.418+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb44112440 con 0x7fcb44103d80 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.422+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcb3402d3f0 con 0x7fcb44103d80 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.618+0000 7fcb42ffd700 1 -- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 msgr2=0x7fcb2c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.618+0000 7fcb42ffd700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 0x7fcb2c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.667+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcb44101290 con 0x7fcb44103d80 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.668+0000 7fcb2bfff700 1 -- 172.21.15.73:0/574963442 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fcb34018350 con 0x7fcb44103d80 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 msgr2=0x7fcb2c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 0x7fcb2c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 msgr2=0x7fcb44111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44111870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb34011650 tx=0x7fcb34009250 comp rx=0 tx=0).stop 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 shutdown_connections 2024-02-10T06:34:44.332 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fcb2c0367e0 0x7fcb2c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:44.333 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 --2- 172.21.15.73:0/574963442 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb44103d80 0x7fcb44111870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:44.333 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 >> 172.21.15.73:0/574963442 conn(0x7fcb440fc580 msgr2=0x7fcb440fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:44.333 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 shutdown_connections 2024-02-10T06:34:44.333 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:43.670+0000 7fcb49fa7700 1 -- 172.21.15.73:0/574963442 wait complete. 2024-02-10T06:34:44.333 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for the mgr to restart... 2024-02-10T06:34:44.333 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr epoch 9... 2024-02-10T06:34:44.557 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:44 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/574963442' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T06:34:49.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: Active manager daemon smithi073.vjivfn restarted 2024-02-10T06:34:49.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: Activating manager daemon smithi073.vjivfn 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: mgrmap e10: smithi073.vjivfn(active, starting, since 0.00308405s) 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr metadata", "who": "smithi073.vjivfn", "id": "smithi073.vjivfn"}]: dispatch 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: Manager daemon smithi073.vjivfn is now available 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:49.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:48 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:50.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:49 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:34:50.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:49 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/trash_purge_schedule"}]: dispatch 2024-02-10T06:34:50.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:49 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:50.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:49 smithi073 ceph-mon[29063]: mgrmap e11: smithi073.vjivfn(active, since 1.00622s) 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0cc3dc700 1 Processor -- start 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0cc3dc700 1 -- start start 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0cc3dc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c41077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0cc3dc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c4103190 con 0x7fc0c4076c00 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0ca178700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c41077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:50.690 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0ca178700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c41077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46632/0 (socket says 172.21.15.73:46632) 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0ca178700 1 -- 172.21.15.73:0/2801552065 learned_addr learned my addr 172.21.15.73:0/2801552065 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.772+0000 7fc0ca178700 1 -- 172.21.15.73:0/2801552065 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c4107ce0 con 0x7fc0c4076c00 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0ca178700 1 --2- 172.21.15.73:0/2801552065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c41077a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00098e0 tx=0x7fc0c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578f45bca01afe66 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0c9176700 1 -- 172.21.15.73:0/2801552065 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0c000b860 con 0x7fc0c4076c00 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0c9176700 1 -- 172.21.15.73:0/2801552065 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0c000be90 con 0x7fc0c4076c00 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0c9176700 1 -- 172.21.15.73:0/2801552065 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0c00103c0 con 0x7fc0c4076c00 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/2801552065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 msgr2=0x7fc0c41077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/2801552065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c41077a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00098e0 tx=0x7fc0c0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/2801552065 shutdown_connections 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/2801552065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c41077a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.773+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/2801552065 >> 172.21.15.73:0/2801552065 conn(0x7fc0c40fcdd0 msgr2=0x7fc0c40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/2801552065 shutdown_connections 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/2801552065 wait complete. 2024-02-10T06:34:50.691 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0cc3dc700 1 Processor -- start 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0cc3dc700 1 -- start start 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0cc3dc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c4113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0cc3dc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c4113ca0 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0ca178700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c4113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0ca178700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c4113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46634/0 (socket says 172.21.15.73:46634) 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.774+0000 7fc0ca178700 1 -- 172.21.15.73:0/1155574317 learned_addr learned my addr 172.21.15.73:0/1155574317 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0ca178700 1 -- 172.21.15.73:0/1155574317 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00094f0 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0ca178700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c4113760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00090b0 tx=0x7fc0c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0c00118f0 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0c0010d10 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0c0024e70 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c4113ea0 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.775+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c4114b40 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.776+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fc0c0024440 con 0x7fc0c4076c00 2024-02-10T06:34:50.692 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.776+0000 7fc0baffd700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.776+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 --> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0b00393a0 con 0x7fc0b00367e0 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.776+0000 7fc0c9977700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 msgr2=0x7fc0b0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.776+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc0c004a650 con 0x7fc0c4076c00 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.776+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.977+0000 7fc0c9977700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 msgr2=0x7fc0b0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:44.977+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:45.378+0000 7fc0c9977700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 msgr2=0x7fc0b0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:45.378+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:46.179+0000 7fc0c9977700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 msgr2=0x7fc0b0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:46.179+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:47.780+0000 7fc0c9977700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 msgr2=0x7fc0b0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1157895579 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:47.780+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:48.822+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fc0c0024740 con 0x7fc0c4076c00 2024-02-10T06:34:50.693 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:48.822+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 msgr2=0x7fc0b0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:48.822+0000 7fc0baffd700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.824+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fc0c004b9f0 con 0x7fc0c4076c00 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.824+0000 7fc0baffd700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc0b0039fd0 0x7fc0b003c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.824+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0b00393a0 con 0x7fc0b0039fd0 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.824+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc0b0039fd0 0x7fc0b003c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.825+0000 7fc0c9977700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc0b0039fd0 0x7fc0b003c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40039d0 tx=0x7fc0b4009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.825+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fc0b00393a0 con 0x7fc0b0039fd0 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.831+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc0c4114330 con 0x7fc0b0039fd0 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.831+0000 7fc0baffd700 1 -- 172.21.15.73:0/1155574317 <== mgr.14162 v2:172.21.15.73:6800/2308737934 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc0c4114330 con 0x7fc0b0039fd0 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc0b0039fd0 msgr2=0x7fc0b003c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc0b0039fd0 0x7fc0b003c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40039d0 tx=0x7fc0b4009350 comp rx=0 tx=0).stop 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 msgr2=0x7fc0c4113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c4113760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00090b0 tx=0x7fc0c0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 shutdown_connections 2024-02-10T06:34:50.694 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0c4076c00 0x7fc0c4113760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc0b0039fd0 0x7fc0b003c3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 --2- 172.21.15.73:0/1155574317 >> [v2:172.21.15.73:6800/1157895579,v1:172.21.15.73:6801/1157895579] conn(0x7fc0b00367e0 0x7fc0b0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 >> 172.21.15.73:0/1155574317 conn(0x7fc0c40fcdd0 msgr2=0x7fc0c40fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 shutdown_connections 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:49.832+0000 7fc0cc3dc700 1 -- 172.21.15.73:0/1155574317 wait complete. 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:mgr epoch 9 is available 2024-02-10T06:34:50.695 INFO:teuthology.orchestra.run.smithi073.stdout:Generating a dashboard self-signed certificate... 2024-02-10T06:34:51.574 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: [10/Feb/2024:06:34:49] ENGINE Bus STARTING 2024-02-10T06:34:51.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: [10/Feb/2024:06:34:49] ENGINE Serving on https://172.21.15.73:7150 2024-02-10T06:34:51.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: [10/Feb/2024:06:34:49] ENGINE Bus STARTED 2024-02-10T06:34:51.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T06:34:51.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T06:34:51.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:51.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:51 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:52 smithi073 ceph-mon[29063]: mgrmap e12: smithi073.vjivfn(active, since 2s) 2024-02-10T06:34:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:52 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:52 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:53.044 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.149+0000 7f3b30625700 1 Processor -- start 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.149+0000 7f3b30625700 1 -- start start 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b30625700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b30625700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281030c0 con 0x7f3b28102770 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2e3c1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2e3c1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35114/0 (socket says 172.21.15.73:35114) 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2e3c1700 1 -- 172.21.15.73:0/3647830412 learned_addr learned my addr 172.21.15.73:0/3647830412 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2e3c1700 1 -- 172.21.15.73:0/3647830412 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28103200 con 0x7f3b28102770 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2e3c1700 1 --2- 172.21.15.73:0/3647830412 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28102b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b180098e0 tx=0x7f3b18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2a4edef49a4c66 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2d3bf700 1 -- 172.21.15.73:0/3647830412 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1800b860 con 0x7f3b28102770 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2d3bf700 1 -- 172.21.15.73:0/3647830412 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b1800be90 con 0x7f3b28102770 2024-02-10T06:34:53.045 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.150+0000 7f3b2d3bf700 1 -- 172.21.15.73:0/3647830412 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b180103c0 con 0x7f3b28102770 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 -- 172.21.15.73:0/3647830412 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 msgr2=0x7f3b28102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 --2- 172.21.15.73:0/3647830412 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28102b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b180098e0 tx=0x7f3b18009250 comp rx=0 tx=0).stop 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 -- 172.21.15.73:0/3647830412 shutdown_connections 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 --2- 172.21.15.73:0/3647830412 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28102b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 -- 172.21.15.73:0/3647830412 >> 172.21.15.73:0/3647830412 conn(0x7f3b280fded0 msgr2=0x7f3b28100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 -- 172.21.15.73:0/3647830412 shutdown_connections 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.151+0000 7f3b30625700 1 -- 172.21.15.73:0/3647830412 wait complete. 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b30625700 1 Processor -- start 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b30625700 1 -- start start 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b30625700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b30625700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28077190 con 0x7f3b28102770 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b2e3c1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b2e3c1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35130/0 (socket says 172.21.15.73:35130) 2024-02-10T06:34:53.046 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b2e3c1700 1 -- 172.21.15.73:0/3232574578 learned_addr learned my addr 172.21.15.73:0/3232574578 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.152+0000 7f3b2e3c1700 1 -- 172.21.15.73:0/3232574578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b180094f0 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.153+0000 7f3b2e3c1700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28078d10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b18011680 tx=0x7f3b18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.153+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b18010890 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.153+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b18010ec0 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.153+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b180243f0 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.153+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28077390 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.153+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b28077820 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.154+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f3b1802d3f0 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.154+0000 7f3b1effd700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f3b140366b0 0x7f3b14038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.154+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3b1804a220 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.154+0000 7f3b2dbc0700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f3b140366b0 0x7f3b14038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.154+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2805f5c0 con 0x7f3b28102770 2024-02-10T06:34:53.047 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.154+0000 7f3b2dbc0700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f3b140366b0 0x7f3b14038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009990 tx=0x7f3b24006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.159+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3b1802b8e0 con 0x7f3b28102770 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.286+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f3b1801a8c0 con 0x7f3b28102770 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.358+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b28077c50 con 0x7f3b140366b0 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.485+0000 7f3b1effd700 1 -- 172.21.15.73:0/3232574578 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3b28077c50 con 0x7f3b140366b0 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f3b140366b0 msgr2=0x7f3b14038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f3b140366b0 0x7f3b14038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009990 tx=0x7f3b24006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 msgr2=0x7f3b28078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28078d10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b18011680 tx=0x7f3b18009250 comp rx=0 tx=0).stop 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 shutdown_connections 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f3b140366b0 0x7f3b14038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 --2- 172.21.15.73:0/3232574578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b28102770 0x7f3b28078d10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 >> 172.21.15.73:0/3232574578 conn(0x7f3b280fded0 msgr2=0x7f3b280ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:53.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 shutdown_connections 2024-02-10T06:34:53.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:51.487+0000 7f3b30625700 1 -- 172.21.15.73:0/3232574578 wait complete. 2024-02-10T06:34:53.049 INFO:teuthology.orchestra.run.smithi073.stdout:Creating initial admin user... 2024-02-10T06:34:53.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:53 smithi073 ceph-mon[29063]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:54.551 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$dQE/PVdWbWyST5OPgxmv2eOTWxLZCfAKDfCFUMT9NqDt6wPIjHA9.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707546893, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T06:34:54.551 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.493+0000 7ff8de6ed700 1 Processor -- start 2024-02-10T06:34:54.551 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.493+0000 7ff8de6ed700 1 -- start start 2024-02-10T06:34:54.551 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8de6ed700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d8105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8de6ed700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d8105880 con 0x7ff8d80714c0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d8105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d8105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35142/0 (socket says 172.21.15.73:35142) 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d7fff700 1 -- 172.21.15.73:0/694517643 learned_addr learned my addr 172.21.15.73:0/694517643 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d7fff700 1 -- 172.21.15.73:0/694517643 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d81059c0 con 0x7ff8d80714c0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d7fff700 1 --2- 172.21.15.73:0/694517643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d8105340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80098e0 tx=0x7ff8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1778a64605f63bf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d6ffd700 1 -- 172.21.15.73:0/694517643 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8c800b860 con 0x7ff8d80714c0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d6ffd700 1 -- 172.21.15.73:0/694517643 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8c800be90 con 0x7ff8d80714c0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.494+0000 7ff8d6ffd700 1 -- 172.21.15.73:0/694517643 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8c80103c0 con 0x7ff8d80714c0 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 -- 172.21.15.73:0/694517643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 msgr2=0x7ff8d8105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 --2- 172.21.15.73:0/694517643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d8105340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80098e0 tx=0x7ff8c8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 -- 172.21.15.73:0/694517643 shutdown_connections 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 --2- 172.21.15.73:0/694517643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d8105340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:54.552 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 -- 172.21.15.73:0/694517643 >> 172.21.15.73:0/694517643 conn(0x7ff8d80fcdd0 msgr2=0x7ff8d80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 -- 172.21.15.73:0/694517643 shutdown_connections 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.495+0000 7ff8de6ed700 1 -- 172.21.15.73:0/694517643 wait complete. 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8de6ed700 1 Processor -- start 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8de6ed700 1 -- start start 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8de6ed700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d81115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8de6ed700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d8111b20 con 0x7ff8d80714c0 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8d7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d81115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8d7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d81115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35158/0 (socket says 172.21.15.73:35158) 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8d7fff700 1 -- 172.21.15.73:0/3299350646 learned_addr learned my addr 172.21.15.73:0/3299350646 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:54.553 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8d7fff700 1 -- 172.21.15.73:0/3299350646 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c80094f0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.496+0000 7ff8d7fff700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d81115e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80094c0 tx=0x7ff8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.497+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8c8010890 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.497+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d8111d20 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.497+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8c8010ec0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.497+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8c80243f0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.497+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d81129c0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.497+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7ff8c802d3f0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.498+0000 7ff8d4ff9700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ff8c00367e0 0x7ff8c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.498+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d805f5c0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.498+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff8c804a3d0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.498+0000 7ff8d77fe700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ff8c00367e0 0x7ff8c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.498+0000 7ff8d77fe700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ff8c00367e0 0x7ff8c0038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009990 tx=0x7ff8cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.502+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff8c802b8e0 con 0x7ff8d80714c0 2024-02-10T06:34:54.554 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.702+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff8d805f3e0 con 0x7ff8c00367e0 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.934+0000 7ff8d4ff9700 1 -- 172.21.15.73:0/3299350646 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ff8d805f3e0 con 0x7ff8c00367e0 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ff8c00367e0 msgr2=0x7ff8c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ff8c00367e0 0x7ff8c0038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009990 tx=0x7ff8cc006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 msgr2=0x7ff8d81115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d81115e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80094c0 tx=0x7ff8c8009250 comp rx=0 tx=0).stop 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 shutdown_connections 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ff8c00367e0 0x7ff8c0038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 --2- 172.21.15.73:0/3299350646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8d80714c0 0x7ff8d81115e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.936+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 >> 172.21.15.73:0/3299350646 conn(0x7ff8d80fcdd0 msgr2=0x7ff8d80fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.937+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 shutdown_connections 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:53.937+0000 7ff8de6ed700 1 -- 172.21.15.73:0/3299350646 wait complete. 2024-02-10T06:34:54.555 INFO:teuthology.orchestra.run.smithi073.stdout:Fetching dashboard port number... 2024-02-10T06:34:55.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:54 smithi073 ceph-mon[29063]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:55.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:54 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:34:55.716 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 8443 2024-02-10T06:34:55.716 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.957+0000 7ffae774e700 1 Processor -- start 2024-02-10T06:34:55.716 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.957+0000 7ffae774e700 1 -- start start 2024-02-10T06:34:55.716 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae774e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:55.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae774e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae01030c0 con 0x7ffae0102770 2024-02-10T06:34:55.719 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae54ea700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae54ea700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35172/0 (socket says 172.21.15.73:35172) 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae54ea700 1 -- 172.21.15.73:0/3020657588 learned_addr learned my addr 172.21.15.73:0/3020657588 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae54ea700 1 -- 172.21.15.73:0/3020657588 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0103200 con 0x7ffae0102770 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffae54ea700 1 --2- 172.21.15.73:0/3020657588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae0102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffad00098e0 tx=0x7ffad0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c09468f39e6575 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffad7fff700 1 -- 172.21.15.73:0/3020657588 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad000b860 con 0x7ffae0102770 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffad7fff700 1 -- 172.21.15.73:0/3020657588 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffad000be90 con 0x7ffae0102770 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.958+0000 7ffad7fff700 1 -- 172.21.15.73:0/3020657588 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad00103c0 con 0x7ffae0102770 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 -- 172.21.15.73:0/3020657588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 msgr2=0x7ffae0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 --2- 172.21.15.73:0/3020657588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae0102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffad00098e0 tx=0x7ffad0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 -- 172.21.15.73:0/3020657588 shutdown_connections 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 --2- 172.21.15.73:0/3020657588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae0102b80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 -- 172.21.15.73:0/3020657588 >> 172.21.15.73:0/3020657588 conn(0x7ffae00fded0 msgr2=0x7ffae0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 -- 172.21.15.73:0/3020657588 shutdown_connections 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 -- 172.21.15.73:0/3020657588 wait complete. 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 Processor -- start 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.959+0000 7ffae774e700 1 -- start start 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae774e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae01115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae774e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae0111b00 con 0x7ffae0102770 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae54ea700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae01115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae54ea700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae01115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35188/0 (socket says 172.21.15.73:35188) 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae54ea700 1 -- 172.21.15.73:0/2049446870 learned_addr learned my addr 172.21.15.73:0/2049446870 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae54ea700 1 -- 172.21.15.73:0/2049446870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00094f0 con 0x7ffae0102770 2024-02-10T06:34:55.721 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae54ea700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae01115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffad0011680 tx=0x7ffad0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad0010890 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffad0010ec0 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad00243f0 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae0111d00 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.960+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae01129a0 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.961+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ffad001a8c0 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.961+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae005f5c0 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.961+0000 7ffad5ffb700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ffacc0368d0 0x7ffacc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.962+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ffad004a4b0 con 0x7ffae0102770 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.962+0000 7ffae4ce9700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ffacc0368d0 0x7ffacc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.962+0000 7ffae4ce9700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ffacc0368d0 0x7ffacc038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009990 tx=0x7ffadc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:55.722 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:54.966+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ffad002b4d0 con 0x7ffae0102770 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.150+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ffae004bed0 con 0x7ffae0102770 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.151+0000 7ffad5ffb700 1 -- 172.21.15.73:0/2049446870 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ffad002b2f0 con 0x7ffae0102770 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ffacc0368d0 msgr2=0x7ffacc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ffacc0368d0 0x7ffacc038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009990 tx=0x7ffadc006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 msgr2=0x7ffae01115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae01115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffad0011680 tx=0x7ffad0009250 comp rx=0 tx=0).stop 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 shutdown_connections 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7ffacc0368d0 0x7ffacc038d80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 --2- 172.21.15.73:0/2049446870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffae0102770 0x7ffae01115c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 >> 172.21.15.73:0/2049446870 conn(0x7ffae00fded0 msgr2=0x7ffae00ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 shutdown_connections 2024-02-10T06:34:55.723 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:55.153+0000 7ffae774e700 1 -- 172.21.15.73:0/2049446870 wait complete. 2024-02-10T06:34:55.724 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:34:55.724 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-02-10T06:34:55.729 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:34:55.729 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-02-10T06:34:55.729 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-02-10T06:34:55.729 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T06:34:55.731 INFO:teuthology.orchestra.run.smithi073.stdout:Ceph Dashboard is now available at: 2024-02-10T06:34:55.731 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:55.731 INFO:teuthology.orchestra.run.smithi073.stdout: URL: https://smithi073.front.sepia.ceph.com:8443/ 2024-02-10T06:34:55.731 INFO:teuthology.orchestra.run.smithi073.stdout: User: admin 2024-02-10T06:34:55.731 INFO:teuthology.orchestra.run.smithi073.stdout: Password: zjli1h2dq6 2024-02-10T06:34:55.731 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:55.732 INFO:teuthology.orchestra.run.smithi073.stdout:Saving cluster configuration to /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config directory 2024-02-10T06:34:55.732 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling autotune for osd_memory_target 2024-02-10T06:34:56.053 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:55 smithi073 ceph-mon[29063]: mgrmap e13: smithi073.vjivfn(active, since 6s) 2024-02-10T06:34:56.053 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:55 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2049446870' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T06:34:56.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.151+0000 7f6e882a2700 1 Processor -- start 2024-02-10T06:34:56.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.151+0000 7f6e882a2700 1 -- start start 2024-02-10T06:34:56.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e882a2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e80101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:56.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e882a2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80101fc0 con 0x7f6e80101670 2024-02-10T06:34:56.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e8603e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e80101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e8603e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e80101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35196/0 (socket says 172.21.15.73:35196) 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e8603e700 1 -- 172.21.15.73:0/123246987 learned_addr learned my addr 172.21.15.73:0/123246987 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e8603e700 1 -- 172.21.15.73:0/123246987 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80102100 con 0x7f6e80101670 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.152+0000 7f6e8603e700 1 --2- 172.21.15.73:0/123246987 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e80101a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00bb80 tx=0x7f6e7c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c00bed1ba4477 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.153+0000 7f6e8503c700 1 -- 172.21.15.73:0/123246987 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e7c00dd00 con 0x7f6e80101670 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.153+0000 7f6e8503c700 1 -- 172.21.15.73:0/123246987 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e7c00de60 con 0x7f6e80101670 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.153+0000 7f6e8503c700 1 -- 172.21.15.73:0/123246987 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e7c018920 con 0x7f6e80101670 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.153+0000 7f6e882a2700 1 -- 172.21.15.73:0/123246987 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 msgr2=0x7f6e80101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.153+0000 7f6e882a2700 1 --2- 172.21.15.73:0/123246987 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e80101a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00bb80 tx=0x7f6e7c00b4f0 comp rx=0 tx=0).stop 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 -- 172.21.15.73:0/123246987 shutdown_connections 2024-02-10T06:34:56.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 --2- 172.21.15.73:0/123246987 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e80101a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 -- 172.21.15.73:0/123246987 >> 172.21.15.73:0/123246987 conn(0x7f6e800fcdd0 msgr2=0x7f6e800ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 -- 172.21.15.73:0/123246987 shutdown_connections 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 -- 172.21.15.73:0/123246987 wait complete. 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 Processor -- start 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.154+0000 7f6e882a2700 1 -- start start 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e882a2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e8010f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e882a2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8010fb40 con 0x7f6e80101670 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e8603e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e8010f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e8603e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e8010f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35210/0 (socket says 172.21.15.73:35210) 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e8603e700 1 -- 172.21.15.73:0/1071235461 learned_addr learned my addr 172.21.15.73:0/1071235461 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e8603e700 1 -- 172.21.15.73:0/1071235461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c00b790 con 0x7f6e80101670 2024-02-10T06:34:56.962 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e8603e700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e8010f600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c007010 tx=0x7f6e7c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e7c010c00 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e7c01d050 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8010fd40 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.155+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e801109e0 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.156+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e7c0226b0 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.156+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6e7c022920 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.156+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e640052e0 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.157+0000 7f6e76ffd700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f6e6c036540 0x7f6e6c0389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.157+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6e7c014070 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.157+0000 7f6e8583d700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f6e6c036540 0x7f6e6c0389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.157+0000 7f6e8583d700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f6e6c036540 0x7f6e6c0389f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009990 tx=0x7f6e70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.161+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6e7c021bf0 con 0x7f6e80101670 2024-02-10T06:34:56.963 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.348+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6e64005b10 con 0x7f6e80101670 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.349+0000 7f6e76ffd700 1 -- 172.21.15.73:0/1071235461 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6e7c0217e0 con 0x7f6e80101670 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f6e6c036540 msgr2=0x7f6e6c0389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f6e6c036540 0x7f6e6c0389f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009990 tx=0x7f6e70006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 msgr2=0x7f6e8010f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e8010f600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c007010 tx=0x7f6e7c00b4f0 comp rx=0 tx=0).stop 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 shutdown_connections 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f6e6c036540 0x7f6e6c0389f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 --2- 172.21.15.73:0/1071235461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e80101670 0x7f6e8010f600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 >> 172.21.15.73:0/1071235461 conn(0x7f6e800fcdd0 msgr2=0x7f6e801063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 shutdown_connections 2024-02-10T06:34:56.964 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:56.351+0000 7f6e882a2700 1 -- 172.21.15.73:0/1071235461 wait complete. 2024-02-10T06:34:58.240 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.378+0000 7fc2fbd74700 1 Processor -- start 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2fbd74700 1 -- start start 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2fbd74700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2fbd74700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f41030c0 con 0x7fc2f4102770 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2f9b10700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2f9b10700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35222/0 (socket says 172.21.15.73:35222) 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2f9b10700 1 -- 172.21.15.73:0/321596254 learned_addr learned my addr 172.21.15.73:0/321596254 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.379+0000 7fc2f9b10700 1 -- 172.21.15.73:0/321596254 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f4103200 con 0x7fc2f4102770 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2f9b10700 1 --2- 172.21.15.73:0/321596254 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f4102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40098e0 tx=0x7fc2e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a0df019ea0eff7 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2f8b0e700 1 -- 172.21.15.73:0/321596254 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e400b860 con 0x7fc2f4102770 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2f8b0e700 1 -- 172.21.15.73:0/321596254 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e400be90 con 0x7fc2f4102770 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2f8b0e700 1 -- 172.21.15.73:0/321596254 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e40103c0 con 0x7fc2f4102770 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 -- 172.21.15.73:0/321596254 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 msgr2=0x7fc2f4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 --2- 172.21.15.73:0/321596254 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f4102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40098e0 tx=0x7fc2e4009250 comp rx=0 tx=0).stop 2024-02-10T06:34:58.241 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 -- 172.21.15.73:0/321596254 shutdown_connections 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 --2- 172.21.15.73:0/321596254 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f4102b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 -- 172.21.15.73:0/321596254 >> 172.21.15.73:0/321596254 conn(0x7fc2f40fded0 msgr2=0x7fc2f4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 -- 172.21.15.73:0/321596254 shutdown_connections 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.380+0000 7fc2fbd74700 1 -- 172.21.15.73:0/321596254 wait complete. 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2fbd74700 1 Processor -- start 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2fbd74700 1 -- start start 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2fbd74700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f41115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2fbd74700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4111b00 con 0x7fc2f4102770 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2f9b10700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f41115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2f9b10700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f41115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35226/0 (socket says 172.21.15.73:35226) 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2f9b10700 1 -- 172.21.15.73:0/3311229169 learned_addr learned my addr 172.21.15.73:0/3311229169 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:34:58.242 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.381+0000 7fc2f9b10700 1 -- 172.21.15.73:0/3311229169 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40094f0 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2f9b10700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f41115c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40090b0 tx=0x7fc2e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e4010840 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e4010e70 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f4111d00 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e4024e40 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f41129a0 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.382+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc2e40109a0 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.383+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f405f5c0 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.383+0000 7fc2ea7fc700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc2e0036880 0x7fc2e0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.383+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc2e404a960 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.383+0000 7fc2f930f700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc2e0036880 0x7fc2e0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.383+0000 7fc2f930f700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc2e0036880 0x7fc2e0038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009990 tx=0x7fc2f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.387+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc2e402e440 con 0x7fc2f4102770 2024-02-10T06:34:58.243 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.669+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fc2f404bed0 con 0x7fc2f4102770 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.671+0000 7fc2ea7fc700 1 -- 172.21.15.73:0/3311229169 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fc2e400f980 con 0x7fc2f4102770 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc2e0036880 msgr2=0x7fc2e0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc2e0036880 0x7fc2e0038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009990 tx=0x7fc2f0006e40 comp rx=0 tx=0).stop 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 msgr2=0x7fc2f41115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f41115c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40090b0 tx=0x7fc2e4009250 comp rx=0 tx=0).stop 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 shutdown_connections 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fc2e0036880 0x7fc2e0038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 --2- 172.21.15.73:0/3311229169 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc2f4102770 0x7fc2f41115c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 >> 172.21.15.73:0/3311229169 conn(0x7fc2f40fded0 msgr2=0x7fc2f40ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 shutdown_connections 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-02-10T06:34:57.673+0000 7fc2fbd74700 1 -- 172.21.15.73:0/3311229169 wait complete. 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-10T06:34:58.244 INFO:teuthology.orchestra.run.smithi073.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout:Or, if you are only running a single cluster on this host: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: ceph telemetry on 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout:For more information see: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T06:34:58.245 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:34:58.246 INFO:teuthology.orchestra.run.smithi073.stdout:Bootstrap complete. 2024-02-10T06:34:58.277 INFO:tasks.cephadm:Fetching config... 2024-02-10T06:34:58.277 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:34:58.277 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T06:34:58.333 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T06:34:58.334 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:34:58.334 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T06:34:58.388 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T06:34:58.388 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:34:58.389 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/keyring of=/dev/stdout 2024-02-10T06:34:58.455 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T06:34:58.456 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:34:58.456 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T06:34:58.510 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T06:34:58.511 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRGpo10GKY2Rwb2E6DI1/+giSgvoiDMBtbYYue7vgUtghsB6LXOMRyDU4fG8zSLod/8jt2/IrupU3gVVLiU/3+GROafpq/W2B/pXHOmFAaW1Df4DGajELTuyqCE8SqTcLOrheXTy/s/ZUL1y4rc1hN38i975ZaBW68EvAk8H2TeLbWaSOslcBjm5a7zKWuE9TnbXXqLNZzZuAk70l/4dUpT5KCiXFUe5qXbDFKPbxq+xEV64fwDF+AXseLel4lgKvXfyzH5ADT6tdIOkQ7pV40yicMtbX3EQq2a1elWuSuQKZBaqHJJo6fl/gcuk/2TqUmrvpTbKMf2V2EGi4d5hsZv1pN1Q3MvObWRpx8Zf9/iTr9nYYmniK6QwvMI3vPS0btZi0t8K1lSvDalSBWT8kKtDff7iKO+2vN4AVE6qsnquwiKebbm+iRnEnqTZRHMgL5/gorRIraSla+yZiRFKv/WXEj5LiojoM0W2GtYyEFNP9yDenvZk4Nc9UFeP05kFU= ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T06:34:58.604 INFO:teuthology.orchestra.run.smithi073.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRGpo10GKY2Rwb2E6DI1/+giSgvoiDMBtbYYue7vgUtghsB6LXOMRyDU4fG8zSLod/8jt2/IrupU3gVVLiU/3+GROafpq/W2B/pXHOmFAaW1Df4DGajELTuyqCE8SqTcLOrheXTy/s/ZUL1y4rc1hN38i975ZaBW68EvAk8H2TeLbWaSOslcBjm5a7zKWuE9TnbXXqLNZzZuAk70l/4dUpT5KCiXFUe5qXbDFKPbxq+xEV64fwDF+AXseLel4lgKvXfyzH5ADT6tdIOkQ7pV40yicMtbX3EQq2a1elWuSuQKZBaqHJJo6fl/gcuk/2TqUmrvpTbKMf2V2EGi4d5hsZv1pN1Q3MvObWRpx8Zf9/iTr9nYYmniK6QwvMI3vPS0btZi0t8K1lSvDalSBWT8kKtDff7iKO+2vN4AVE6qsnquwiKebbm+iRnEnqTZRHMgL5/gorRIraSla+yZiRFKv/WXEj5LiojoM0W2GtYyEFNP9yDenvZk4Nc9UFeP05kFU= ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:34:58.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRGpo10GKY2Rwb2E6DI1/+giSgvoiDMBtbYYue7vgUtghsB6LXOMRyDU4fG8zSLod/8jt2/IrupU3gVVLiU/3+GROafpq/W2B/pXHOmFAaW1Df4DGajELTuyqCE8SqTcLOrheXTy/s/ZUL1y4rc1hN38i975ZaBW68EvAk8H2TeLbWaSOslcBjm5a7zKWuE9TnbXXqLNZzZuAk70l/4dUpT5KCiXFUe5qXbDFKPbxq+xEV64fwDF+AXseLel4lgKvXfyzH5ADT6tdIOkQ7pV40yicMtbX3EQq2a1elWuSuQKZBaqHJJo6fl/gcuk/2TqUmrvpTbKMf2V2EGi4d5hsZv1pN1Q3MvObWRpx8Zf9/iTr9nYYmniK6QwvMI3vPS0btZi0t8K1lSvDalSBWT8kKtDff7iKO+2vN4AVE6qsnquwiKebbm+iRnEnqTZRHMgL5/gorRIraSla+yZiRFKv/WXEj5LiojoM0W2GtYyEFNP9yDenvZk4Nc9UFeP05kFU= ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T06:34:58.664 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRGpo10GKY2Rwb2E6DI1/+giSgvoiDMBtbYYue7vgUtghsB6LXOMRyDU4fG8zSLod/8jt2/IrupU3gVVLiU/3+GROafpq/W2B/pXHOmFAaW1Df4DGajELTuyqCE8SqTcLOrheXTy/s/ZUL1y4rc1hN38i975ZaBW68EvAk8H2TeLbWaSOslcBjm5a7zKWuE9TnbXXqLNZzZuAk70l/4dUpT5KCiXFUe5qXbDFKPbxq+xEV64fwDF+AXseLel4lgKvXfyzH5ADT6tdIOkQ7pV40yicMtbX3EQq2a1elWuSuQKZBaqHJJo6fl/gcuk/2TqUmrvpTbKMf2V2EGi4d5hsZv1pN1Q3MvObWRpx8Zf9/iTr9nYYmniK6QwvMI3vPS0btZi0t8K1lSvDalSBWT8kKtDff7iKO+2vN4AVE6qsnquwiKebbm+iRnEnqTZRHMgL5/gorRIraSla+yZiRFKv/WXEj5LiojoM0W2GtYyEFNP9yDenvZk4Nc9UFeP05kFU= ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:34:58.679 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T06:34:58.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:34:58 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3311229169' entity='client.admin' 2024-02-10T06:34:58.931 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:35:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 -- 172.21.15.73:0/3303516264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 msgr2=0x7f75600ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 --2- 172.21.15.73:0/3303516264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600ffaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f755c0048c0 tx=0x7f755c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 -- 172.21.15.73:0/3303516264 shutdown_connections 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 --2- 172.21.15.73:0/3303516264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600ffaf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 -- 172.21.15.73:0/3303516264 >> 172.21.15.73:0/3303516264 conn(0x7f75600f7190 msgr2=0x7f75600f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 -- 172.21.15.73:0/3303516264 shutdown_connections 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.215+0000 7f75681a1700 1 -- 172.21.15.73:0/3303516264 wait complete. 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75681a1700 1 Processor -- start 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75681a1700 1 -- start start 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75681a1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75681a1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600fb620 con 0x7f75600fd710 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f7565f3d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f7565f3d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600fd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49586/0 (socket says 172.21.15.73:49586) 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f7565f3d700 1 -- 172.21.15.73:0/1079369342 learned_addr learned my addr 172.21.15.73:0/1079369342 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:35:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f7565f3d700 1 -- 172.21.15.73:0/1079369342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c005040 con 0x7f75600fd710 2024-02-10T06:35:01.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f7565f3d700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600fd1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f755c009680 tx=0x7f755c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:01.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755c00b5c0 con 0x7f75600fd710 2024-02-10T06:35:01.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f755c004e60 con 0x7f75600fd710 2024-02-10T06:35:01.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755c01c590 con 0x7f75600fd710 2024-02-10T06:35:01.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75600fb820 con 0x7f75600fd710 2024-02-10T06:35:01.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.216+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75600fbd10 con 0x7f75600fd710 2024-02-10T06:35:01.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.217+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f755c013420 con 0x7f75600fd710 2024-02-10T06:35:01.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.217+0000 7f75567fc700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f754c0368c0 0x7f754c038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:01.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.217+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75601069a0 con 0x7f75600fd710 2024-02-10T06:35:01.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.217+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f755c049ab0 con 0x7f75600fd710 2024-02-10T06:35:01.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.218+0000 7f756573c700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f754c0368c0 0x7f754c038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:01.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.218+0000 7f756573c700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f754c0368c0 0x7f754c038d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7550009990 tx=0x7f7550006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:01.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.222+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f755c025400 con 0x7f75600fd710 2024-02-10T06:35:01.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.409+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f756005f5c0 con 0x7f75600fd710 2024-02-10T06:35:01.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.415+0000 7f75567fc700 1 -- 172.21.15.73:0/1079369342 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f755c02bb10 con 0x7f75600fd710 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f754c0368c0 msgr2=0x7f754c038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f754c0368c0 0x7f754c038d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7550009990 tx=0x7f7550006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 msgr2=0x7f75600fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600fd1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f755c009680 tx=0x7f755c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 shutdown_connections 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f754c0368c0 0x7f754c038d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 --2- 172.21.15.73:0/1079369342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75600fd710 0x7f75600fd1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 >> 172.21.15.73:0/1079369342 conn(0x7f75600f7190 msgr2=0x7f75600f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:01.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 shutdown_connections 2024-02-10T06:35:01.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:01.417+0000 7f75681a1700 1 -- 172.21.15.73:0/1079369342 wait complete. 2024-02-10T06:35:01.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:01.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:01.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:35:01.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:01.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:35:01.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:35:01.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:35:01.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:01 smithi073 ceph-mon[29063]: Deploying daemon crash.smithi073 on smithi073 2024-02-10T06:35:02.096 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T06:35:02.097 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T06:35:02.451 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:35:02.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:02 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1079369342' entity='client.admin' 2024-02-10T06:35:05.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 -- 172.21.15.73:0/2364006258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 msgr2=0x7f2b680fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:05.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 --2- 172.21.15.73:0/2364006258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b680fcfd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009c60 tx=0x7f2b60004d10 comp rx=0 tx=0).stop 2024-02-10T06:35:05.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 -- 172.21.15.73:0/2364006258 shutdown_connections 2024-02-10T06:35:05.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 --2- 172.21.15.73:0/2364006258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b680fcfd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:05.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 -- 172.21.15.73:0/2364006258 >> 172.21.15.73:0/2364006258 conn(0x7f2b680f82a0 msgr2=0x7f2b680fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:05.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 -- 172.21.15.73:0/2364006258 shutdown_connections 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.717+0000 7f2b6fd61700 1 -- 172.21.15.73:0/2364006258 wait complete. 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6fd61700 1 Processor -- start 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6fd61700 1 -- start start 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6fd61700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b6810ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6fd61700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6810d3f0 con 0x7f2b680fcbc0 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6dafd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b6810ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6dafd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b6810ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49608/0 (socket says 172.21.15.73:49608) 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6dafd700 1 -- 172.21.15.73:0/3631634560 learned_addr learned my addr 172.21.15.73:0/3631634560 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:35:05.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6dafd700 1 -- 172.21.15.73:0/3631634560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b60009940 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6dafd700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b6810ceb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2b60015040 tx=0x7f2b60004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b60007c80 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b6001d910 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6810d5f0 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.718+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b600129f0 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.719+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6810e290 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.719+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f2b60019040 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.719+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6805f5c0 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.719+0000 7f2b5e7fc700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f2b54036870 0x7f2b54038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.720+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2b6004a3a0 con 0x7f2b680fcbc0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.720+0000 7f2b6d2fc700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f2b54036870 0x7f2b54038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:05.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.720+0000 7f2b6d2fc700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f2b54036870 0x7f2b54038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009990 tx=0x7f2b58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:05.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.724+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2b60024310 con 0x7f2b680fcbc0 2024-02-10T06:35:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.925+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b6810edf0 con 0x7f2b54036870 2024-02-10T06:35:05.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.928+0000 7f2b5e7fc700 1 -- 172.21.15.73:0/3631634560 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2b6810edf0 con 0x7f2b54036870 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f2b54036870 msgr2=0x7f2b54038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f2b54036870 0x7f2b54038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009990 tx=0x7f2b58006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 msgr2=0x7f2b6810ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b6810ceb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2b60015040 tx=0x7f2b60004d10 comp rx=0 tx=0).stop 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 shutdown_connections 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f2b54036870 0x7f2b54038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 --2- 172.21.15.73:0/3631634560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b680fcbc0 0x7f2b6810ceb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:05.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 >> 172.21.15.73:0/3631634560 conn(0x7f2b680f82a0 msgr2=0x7f2b680f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:05.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 shutdown_connections 2024-02-10T06:35:05.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:05.930+0000 7f2b6fd61700 1 -- 172.21.15.73:0/3631634560 wait complete. 2024-02-10T06:35:06.942 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-02-10T06:35:06.942 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:35:06.942 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-02-10T06:35:06.961 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:35:06.962 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:35:07.019 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-02-10T06:35:07.020 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch host add smithi181 2024-02-10T06:35:07.286 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:06 smithi073 ceph-mon[29063]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:07.286 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:06 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:07.286 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:06 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:07.286 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:06 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:07.286 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:06 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:07.286 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:06 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:07.299 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:35:08.248 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:07 smithi073 ceph-mon[29063]: Deploying daemon node-exporter.smithi073 on smithi073 2024-02-10T06:35:09.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 -- 172.21.15.73:0/2871037573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 msgr2=0x7f31cc0fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 --2- 172.21.15.73:0/2871037573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc0fbee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009c60 tx=0x7f31c8004d10 comp rx=0 tx=0).stop 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 -- 172.21.15.73:0/2871037573 shutdown_connections 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 --2- 172.21.15.73:0/2871037573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc0fbee0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 -- 172.21.15.73:0/2871037573 >> 172.21.15.73:0/2871037573 conn(0x7f31cc0f7190 msgr2=0x7f31cc0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 -- 172.21.15.73:0/2871037573 shutdown_connections 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 -- 172.21.15.73:0/2871037573 wait complete. 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 Processor -- start 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.124+0000 7f31d2e40700 1 -- start start 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d2e40700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d2e40700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc102250 con 0x7f31cc0fbad0 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d0bdc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d0bdc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49096/0 (socket says 172.21.15.73:49096) 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d0bdc700 1 -- 172.21.15.73:0/1819088951 learned_addr learned my addr 172.21.15.73:0/1819088951 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:35:09.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d0bdc700 1 -- 172.21.15.73:0/1819088951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c8009940 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d0bdc700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc103dd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f31c8015040 tx=0x7f31c8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31c8007c80 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31c801d910 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc102450 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31c80129f0 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.125+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc102940 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.126+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f31c8019040 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.127+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc04bed0 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.127+0000 7f31c17fa700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f31b40368c0 0x7f31b4038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.127+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f31c804a490 con 0x7f31cc0fbad0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.127+0000 7f31c3fff700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f31b40368c0 0x7f31b4038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.127+0000 7f31c3fff700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f31b40368c0 0x7f31b4038d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009990 tx=0x7f31bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:09.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.131+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f31c8024310 con 0x7f31cc0fbad0 2024-02-10T06:35:09.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:09.326+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}) v1 -- 0x7f31cc103060 con 0x7f31b40368c0 2024-02-10T06:35:10.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:09 smithi073 ceph-mon[29063]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:10.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:09 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:10.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:09 smithi073 ceph-mon[29063]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:10 smithi073 ceph-mon[29063]: Deploying cephadm binary to smithi181 2024-02-10T06:35:11.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.408+0000 7f31c17fa700 1 -- 172.21.15.73:0/1819088951 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f31cc103060 con 0x7f31b40368c0 2024-02-10T06:35:11.410 INFO:teuthology.orchestra.run.smithi073.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-02-10T06:35:11.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.410+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f31b40368c0 msgr2=0x7f31b4038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.410+0000 7f31d2e40700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f31b40368c0 0x7f31b4038d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009990 tx=0x7f31bc006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.410+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 msgr2=0x7f31cc103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.410+0000 7f31d2e40700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc103dd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f31c8015040 tx=0x7f31c8004d10 comp rx=0 tx=0).stop 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.411+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 shutdown_connections 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.411+0000 7f31d2e40700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f31b40368c0 0x7f31b4038d70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.411+0000 7f31d2e40700 1 --2- 172.21.15.73:0/1819088951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31cc0fbad0 0x7f31cc103dd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.411+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 >> 172.21.15.73:0/1819088951 conn(0x7f31cc0f7190 msgr2=0x7f31cc0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.411+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 shutdown_connections 2024-02-10T06:35:11.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:11.411+0000 7f31d2e40700 1 -- 172.21.15.73:0/1819088951 wait complete. 2024-02-10T06:35:12.128 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T06:35:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:12 smithi073 ceph-mon[29063]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:12 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:12 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:12 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:12 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:12 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:35:13.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:13 smithi073 ceph-mon[29063]: Added host smithi181 2024-02-10T06:35:13.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:13 smithi073 ceph-mon[29063]: Deploying daemon alertmanager.smithi073 on smithi073 2024-02-10T06:35:14.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:14 smithi073 ceph-mon[29063]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:14.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:14 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 -- 172.21.15.73:0/3846666788 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 msgr2=0x7fa5280fd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 --2- 172.21.15.73:0/3846666788 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa5280fd550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5180048c0 tx=0x7fa518009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 -- 172.21.15.73:0/3846666788 shutdown_connections 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 --2- 172.21.15.73:0/3846666788 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa5280fd550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 -- 172.21.15.73:0/3846666788 >> 172.21.15.73:0/3846666788 conn(0x7fa5280f6920 msgr2=0x7fa5280f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 -- 172.21.15.73:0/3846666788 shutdown_connections 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.716+0000 7fa52e626700 1 -- 172.21.15.73:0/3846666788 wait complete. 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa52e626700 1 Processor -- start 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa52e626700 1 -- start start 2024-02-10T06:35:14.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa52e626700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa52810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa52e626700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52810d410 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa527fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa52810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa527fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa52810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49114/0 (socket says 172.21.15.73:49114) 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.717+0000 7fa527fff700 1 -- 172.21.15.73:0/884795543 learned_addr learned my addr 172.21.15.73:0/884795543 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa527fff700 1 -- 172.21.15.73:0/884795543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518005040 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa527fff700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa52810ced0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa518009b80 tx=0x7fa518009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51800b5c0 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa51800be70 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51801c660 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52810d610 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.718+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52810e2b0 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.719+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fa518004d30 con 0x7fa5280fd140 2024-02-10T06:35:14.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.719+0000 7fa524ff9700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa510036870 0x7fa510038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:14.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.719+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52805f5c0 con 0x7fa5280fd140 2024-02-10T06:35:14.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.719+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa518049a30 con 0x7fa5280fd140 2024-02-10T06:35:14.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.719+0000 7fa5277fe700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa510036870 0x7fa510038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:14.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.719+0000 7fa5277fe700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa510036870 0x7fa510038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009990 tx=0x7fa51c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:14.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.724+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa518025400 con 0x7fa5280fd140 2024-02-10T06:35:14.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.924+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa52810ee70 con 0x7fa510036870 2024-02-10T06:35:14.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.925+0000 7fa524ff9700 1 -- 172.21.15.73:0/884795543 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fa52810ee70 con 0x7fa510036870 2024-02-10T06:35:14.926 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:35:14.926 INFO:teuthology.orchestra.run.smithi073.stdout:[{"addr": "172.21.15.73", "hostname": "smithi073", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa510036870 msgr2=0x7fa510038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa510036870 0x7fa510038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009990 tx=0x7fa51c006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 msgr2=0x7fa52810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa52810ced0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa518009b80 tx=0x7fa518009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 shutdown_connections 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa510036870 0x7fa510038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 --2- 172.21.15.73:0/884795543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5280fd140 0x7fa52810ced0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 >> 172.21.15.73:0/884795543 conn(0x7fa5280f6920 msgr2=0x7fa5280fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 shutdown_connections 2024-02-10T06:35:14.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:14.927+0000 7fa52e626700 1 -- 172.21.15.73:0/884795543 wait complete. 2024-02-10T06:35:15.905 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T06:35:15.906 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T06:35:16.158 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:35:16.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:16 smithi073 ceph-mon[29063]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:16.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:16 smithi073 ceph-mon[29063]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:35:17.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/3848365389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 msgr2=0x7f7cc80fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:17.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 --2- 172.21.15.73:0/3848365389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc80fbec0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40048c0 tx=0x7f7cc4009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:17.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/3848365389 shutdown_connections 2024-02-10T06:35:17.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 --2- 172.21.15.73:0/3848365389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc80fbec0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:17.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/3848365389 >> 172.21.15.73:0/3848365389 conn(0x7f7cc80f7210 msgr2=0x7f7cc80f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/3848365389 shutdown_connections 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.953+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/3848365389 wait complete. 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccfc3e700 1 Processor -- start 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccfc3e700 1 -- start start 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccfc3e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc8111fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccfc3e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc810d9b0 con 0x7f7cc80fbab0 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccd9da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc8111fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccd9da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc8111fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49142/0 (socket says 172.21.15.73:49142) 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccd9da700 1 -- 172.21.15.73:0/790061040 learned_addr learned my addr 172.21.15.73:0/790061040 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.954+0000 7f7ccd9da700 1 -- 172.21.15.73:0/790061040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc4005040 con 0x7f7cc80fbab0 2024-02-10T06:35:17.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.955+0000 7f7ccd9da700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc8111fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009b80 tx=0x7f7cc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.955+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc400b5c0 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.955+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cc400be70 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.955+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc8066830 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.955+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc401c660 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.955+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc8112500 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.956+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f7cc4013420 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.956+0000 7f7cbe7fc700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7cb4036870 0x7f7cb4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.956+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7cc4049a10 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.956+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc8105800 con 0x7f7cc80fbab0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.956+0000 7f7ccd1d9700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7cb4036870 0x7f7cb4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:17.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.956+0000 7f7ccd1d9700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7cb4036870 0x7f7cb4038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009990 tx=0x7f7cb8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:17.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:17.961+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7cc4013770 con 0x7f7cc80fbab0 2024-02-10T06:35:18.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.167+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7cc804bed0 con 0x7f7cc80fbab0 2024-02-10T06:35:18.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.268+0000 7f7cbe7fc700 1 -- 172.21.15.73:0/790061040 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7cc4011370 con 0x7f7cc80fbab0 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7cb4036870 msgr2=0x7f7cb4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7cb4036870 0x7f7cb4038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009990 tx=0x7f7cb8006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 msgr2=0x7f7cc8111fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc8111fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009b80 tx=0x7f7cc4009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 shutdown_connections 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7cb4036870 0x7f7cb4038d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:18.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 --2- 172.21.15.73:0/790061040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7cc80fbab0 0x7f7cc8111fc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:18.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 >> 172.21.15.73:0/790061040 conn(0x7f7cc80f7210 msgr2=0x7f7cc80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:18.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 shutdown_connections 2024-02-10T06:35:18.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:35:18.270+0000 7f7ccfc3e700 1 -- 172.21.15.73:0/790061040 wait complete. 2024-02-10T06:35:18.272 INFO:teuthology.orchestra.run.smithi073.stderr:adjusted tunables profile to default 2024-02-10T06:35:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:18 smithi073 ceph-mon[29063]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:18 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/790061040' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T06:35:19.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:19 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/790061040' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T06:35:19.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:19 smithi073 ceph-mon[29063]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T06:35:19.613 INFO:tasks.cephadm:Adding mon.smithi073 on smithi073 2024-02-10T06:35:19.613 INFO:tasks.cephadm:Adding mon.smithi181 on smithi181 2024-02-10T06:35:19.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi073:172.21.15.73=smithi073;smithi181:172.21.15.181=smithi181' 2024-02-10T06:35:19.861 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:19.908 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:20.369 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:20 smithi073 ceph-mon[29063]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:21.620 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/735239761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 msgr2=0x7f0fbc0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:21.620 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 --2- 172.21.15.181:0/735239761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc0fbed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0048c0 tx=0x7f0fac009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/735239761 shutdown_connections 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 --2- 172.21.15.181:0/735239761 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc0fbed0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/735239761 >> 172.21.15.181:0/735239761 conn(0x7f0fbc0f71a0 msgr2=0x7f0fbc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/735239761 shutdown_connections 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/735239761 wait complete. 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 Processor -- start 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- start start 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc111fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.618+0000 7f0fc3fd9700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc10d9d0 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc1d75700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc111fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc1d75700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc111fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:37404/0 (socket says 172.21.15.181:37404) 2024-02-10T06:35:21.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc1d75700 1 -- 172.21.15.181:0/1456283911 learned_addr learned my addr 172.21.15.181:0/1456283911 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc1d75700 1 -- 172.21.15.181:0/1456283911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac005040 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc1d75700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc111fe0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009680 tx=0x7f0fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fac00bed0 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fbc0665f0 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fac00b5c0 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fac01c5b0 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.619+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc112520 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.620+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fbc105890 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.620+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f0fac013a80 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.620+0000 7f0fb27fc700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f0fa8036820 0x7f0fa8038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:21.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.620+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0fac049b70 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.621+0000 7f0fc1574700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f0fa8036820 0x7f0fa8038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:21.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.621+0000 7f0fc1574700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f0fa8036820 0x7f0fa8038cd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009990 tx=0x7f0fb8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:21.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.625+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0fac011940 con 0x7f0fbc0fbac0 2024-02-10T06:35:21.815 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.814+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 --> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi073:172.21.15.73=smithi073;smithi181:172.21.15.181=smithi181", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fbc02b4d0 con 0x7f0fa8036820 2024-02-10T06:35:21.823 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.822+0000 7f0fb27fc700 1 -- 172.21.15.181:0/1456283911 <== mgr.14162 v2:172.21.15.73:6800/2308737934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0fbc02b4d0 con 0x7f0fa8036820 2024-02-10T06:35:21.823 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f0fa8036820 msgr2=0x7f0fa8038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f0fa8036820 0x7f0fa8038cd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009990 tx=0x7f0fb8006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 msgr2=0x7f0fbc111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc111fe0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009680 tx=0x7f0fac009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 shutdown_connections 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f0fa8036820 0x7f0fa8038cd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 --2- 172.21.15.181:0/1456283911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0fbc0fbac0 0x7f0fbc111fe0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 >> 172.21.15.181:0/1456283911 conn(0x7f0fbc0f71a0 msgr2=0x7f0fbc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 shutdown_connections 2024-02-10T06:35:21.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:21.824+0000 7f0fc3fd9700 1 -- 172.21.15.181:0/1456283911 wait complete. 2024-02-10T06:35:22.520 DEBUG:teuthology.orchestra.run.smithi181:mon.smithi181> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi181.service 2024-02-10T06:35:22.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:22.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:22.540 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:22 smithi073 ceph-mon[29063]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:22.540 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:22 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:22.859 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:22.906 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:23.480 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:23 smithi073 ceph-mon[29063]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi073:172.21.15.73=smithi073;smithi181:172.21.15.181=smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:23.480 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:23 smithi073 ceph-mon[29063]: Saving service mon spec with placement smithi073:172.21.15.73=smithi073;smithi181:172.21.15.181=smithi181;count:2 2024-02-10T06:35:24.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 -- 172.21.15.181:0/3325739216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 msgr2=0x7f38d40ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:24.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 --2- 172.21.15.181:0/3325739216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d40ff7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0048c0 tx=0x7f38cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:24.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 -- 172.21.15.181:0/3325739216 shutdown_connections 2024-02-10T06:35:24.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 --2- 172.21.15.181:0/3325739216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d40ff7e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:24.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 -- 172.21.15.181:0/3325739216 >> 172.21.15.181:0/3325739216 conn(0x7f38d40f7190 msgr2=0x7f38d40f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 -- 172.21.15.181:0/3325739216 shutdown_connections 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.337+0000 7f38db60d700 1 -- 172.21.15.181:0/3325739216 wait complete. 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38db60d700 1 Processor -- start 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38db60d700 1 -- start start 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38db60d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d410cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38db60d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d410d3e0 con 0x7f38d4065d90 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38d93a9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d410cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38d93a9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d410cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:39128/0 (socket says 172.21.15.181:39128) 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38d93a9700 1 -- 172.21.15.181:0/1139623750 learned_addr learned my addr 172.21.15.181:0/1139623750 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38d93a9700 1 -- 172.21.15.181:0/1139623750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc005040 con 0x7f38d4065d90 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.338+0000 7f38d93a9700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d410cea0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009b80 tx=0x7f38cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.339+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38cc00b5c0 con 0x7f38d4065d90 2024-02-10T06:35:24.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.339+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d410d5e0 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.339+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38cc00be70 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.339+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38cc01c660 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.339+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d410e280 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.340+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f38cc01c8d0 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.340+0000 7f38c9ffb700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f38c0036870 0x7f38c0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.340+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f38cc049a60 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.340+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d405f5c0 con 0x7f38d4065d90 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.340+0000 7f38d8ba8700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f38c0036870 0x7f38c0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:24.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.340+0000 7f38d8ba8700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f38c0036870 0x7f38c0038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009990 tx=0x7f38c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:24.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.345+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f38cc011db0 con 0x7f38d4065d90 2024-02-10T06:35:24.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.596+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38d410ed30 con 0x7f38d4065d90 2024-02-10T06:35:24.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.597+0000 7f38c9ffb700 1 -- 172.21.15.181:0/1139623750 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f38cc034090 con 0x7f38d4065d90 2024-02-10T06:35:24.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:24.598 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f38c0036870 msgr2=0x7f38c0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f38c0036870 0x7f38c0038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009990 tx=0x7f38c4006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 msgr2=0x7f38d410cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d410cea0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009b80 tx=0x7f38cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 shutdown_connections 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f38c0036870 0x7f38c0038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:24.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 --2- 172.21.15.181:0/1139623750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38d4065d90 0x7f38d410cea0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:24.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 >> 172.21.15.181:0/1139623750 conn(0x7f38d40f7190 msgr2=0x7f38d40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:24.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 shutdown_connections 2024-02-10T06:35:24.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:24.599+0000 7f38db60d700 1 -- 172.21.15.181:0/1139623750 wait complete. 2024-02-10T06:35:24.601 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:24.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:24.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:35:24.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:24 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:25.539 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:25 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:35:25.539 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:25 smithi073 ceph-mon[29063]: Deploying daemon grafana.smithi073 on smithi073 2024-02-10T06:35:25.539 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:25 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1139623750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:26.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:26.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:26.482 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:26.530 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:26.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:26 smithi073 ceph-mon[29063]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 -- 172.21.15.181:0/1868656608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 msgr2=0x7f1f100fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 --2- 172.21.15.181:0/1868656608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f100fbee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009c60 tx=0x7f1f08004d10 comp rx=0 tx=0).stop 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 -- 172.21.15.181:0/1868656608 shutdown_connections 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 --2- 172.21.15.181:0/1868656608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f100fbee0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 -- 172.21.15.181:0/1868656608 >> 172.21.15.181:0/1868656608 conn(0x7f1f100f7190 msgr2=0x7f1f100f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 -- 172.21.15.181:0/1868656608 shutdown_connections 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.833+0000 7f1f18469700 1 -- 172.21.15.181:0/1868656608 wait complete. 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f18469700 1 Processor -- start 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f18469700 1 -- start start 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f18469700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f1010cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:27.836 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f18469700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1010d400 con 0x7f1f100fbad0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f16205700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f1010cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f16205700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f1010cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:39154/0 (socket says 172.21.15.181:39154) 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f16205700 1 -- 172.21.15.181:0/3544680231 learned_addr learned my addr 172.21.15.181:0/3544680231 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.834+0000 7f1f16205700 1 -- 172.21.15.181:0/3544680231 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08009940 con 0x7f1f100fbad0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.835+0000 7f1f16205700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f1010cec0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1f08015040 tx=0x7f1f08004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.835+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f08007c80 con 0x7f1f100fbad0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.835+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f0801d910 con 0x7f1f100fbad0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.835+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1010d600 con 0x7f1f100fbad0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.835+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f080129f0 con 0x7f1f100fbad0 2024-02-10T06:35:27.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.835+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1010e2a0 con 0x7f1f100fbad0 2024-02-10T06:35:27.838 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.836+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1f08019040 con 0x7f1f100fbad0 2024-02-10T06:35:27.838 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.836+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1005f5c0 con 0x7f1f100fbad0 2024-02-10T06:35:27.838 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.836+0000 7f1f0effd700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1ef80368c0 0x7f1ef8038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:27.838 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.836+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1f0804a420 con 0x7f1f100fbad0 2024-02-10T06:35:27.838 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.836+0000 7f1f15a04700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1ef80368c0 0x7f1ef8038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:27.838 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.837+0000 7f1f15a04700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1ef80368c0 0x7f1ef8038d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009990 tx=0x7f1f04006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:27.842 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:27.841+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1f0802eb40 con 0x7f1f100fbad0 2024-02-10T06:35:28.097 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.096+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f1010ed90 con 0x7f1f100fbad0 2024-02-10T06:35:28.098 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.097+0000 7f1f0effd700 1 -- 172.21.15.181:0/3544680231 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1f08017070 con 0x7f1f100fbad0 2024-02-10T06:35:28.098 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:28.098 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1ef80368c0 msgr2=0x7f1ef8038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1ef80368c0 0x7f1ef8038d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009990 tx=0x7f1f04006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 msgr2=0x7f1f1010cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f1010cec0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1f08015040 tx=0x7f1f08004d10 comp rx=0 tx=0).stop 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 shutdown_connections 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1ef80368c0 0x7f1ef8038d70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:28.100 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 --2- 172.21.15.181:0/3544680231 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f100fbad0 0x7f1f1010cec0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:28.101 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 >> 172.21.15.181:0/3544680231 conn(0x7f1f100f7190 msgr2=0x7f1f10065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:28.101 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 shutdown_connections 2024-02-10T06:35:28.101 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:28.099+0000 7f1f18469700 1 -- 172.21.15.181:0/3544680231 wait complete. 2024-02-10T06:35:28.101 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:28 smithi073 ceph-mon[29063]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:28 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3544680231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:29.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:29.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:29.916 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:29.963 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:30.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:30 smithi073 ceph-mon[29063]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:31.326 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 -- 172.21.15.181:0/2648360295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 msgr2=0x7f1f6c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 --2- 172.21.15.181:0/2648360295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c0fcfe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0048c0 tx=0x7f1f5c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 -- 172.21.15.181:0/2648360295 shutdown_connections 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 --2- 172.21.15.181:0/2648360295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c0fcfe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 -- 172.21.15.181:0/2648360295 >> 172.21.15.181:0/2648360295 conn(0x7f1f6c0f8270 msgr2=0x7f1f6c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 -- 172.21.15.181:0/2648360295 shutdown_connections 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 -- 172.21.15.181:0/2648360295 wait complete. 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 Processor -- start 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.325+0000 7f1f72218700 1 -- start start 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f72218700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f72218700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c10d410 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f6b7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f6b7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c10ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:39184/0 (socket says 172.21.15.181:39184) 2024-02-10T06:35:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f6b7fe700 1 -- 172.21.15.181:0/1509189213 learned_addr learned my addr 172.21.15.181:0/1509189213 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:31.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f6b7fe700 1 -- 172.21.15.181:0/1509189213 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c005040 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f6b7fe700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c10ced0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009680 tx=0x7f1f5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5c00bed0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c10d610 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5c00b5c0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5c01c5b0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.326+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c10e2b0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.327+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1f5c01c7d0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.327+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c05f5c0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.327+0000 7f1f53fff700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1f54036870 0x7f1f54038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.327+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1f5c049c00 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.330 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.328+0000 7f1f6affd700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1f54036870 0x7f1f54038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:31.330 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.328+0000 7f1f6affd700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1f54036870 0x7f1f54038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009990 tx=0x7f1f60006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:31.333 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.333+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1f5c011dd0 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.596 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.592+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f6c10ee00 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.596 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.593+0000 7f1f53fff700 1 -- 172.21.15.181:0/1509189213 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1f5c049e60 con 0x7f1f6c0fcbd0 2024-02-10T06:35:31.596 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:31.596 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:31.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1f54036870 msgr2=0x7f1f54038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:31.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1f54036870 0x7f1f54038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009990 tx=0x7f1f60006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:31.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 msgr2=0x7f1f6c10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:31.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c10ced0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009680 tx=0x7f1f5c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:31.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 shutdown_connections 2024-02-10T06:35:31.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f1f54036870 0x7f1f54038d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:31.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 --2- 172.21.15.181:0/1509189213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f6c0fcbd0 0x7f1f6c10ced0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:31.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 >> 172.21.15.181:0/1509189213 conn(0x7f1f6c0f8270 msgr2=0x7f1f6c0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:31.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 shutdown_connections 2024-02-10T06:35:31.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:31.595+0000 7f1f72218700 1 -- 172.21.15.181:0/1509189213 wait complete. 2024-02-10T06:35:31.598 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:32.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:32 smithi073 ceph-mon[29063]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:32.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:32 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1509189213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:33.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:33.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:33.408 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:33.456 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:34.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:34 smithi073 ceph-mon[29063]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:34.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 -- 172.21.15.181:0/1144073027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 msgr2=0x7fbd4c0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:34.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 --2- 172.21.15.181:0/1144073027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c0fbed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbd400048c0 tx=0x7fbd40009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 -- 172.21.15.181:0/1144073027 shutdown_connections 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 --2- 172.21.15.181:0/1144073027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c0fbed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 -- 172.21.15.181:0/1144073027 >> 172.21.15.181:0/1144073027 conn(0x7fbd4c0f71a0 msgr2=0x7fbd4c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 -- 172.21.15.181:0/1144073027 shutdown_connections 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 -- 172.21.15.181:0/1144073027 wait complete. 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 Processor -- start 2024-02-10T06:35:34.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.854+0000 7fbd52e4e700 1 -- start start 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.855+0000 7fbd52e4e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c10fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.855+0000 7fbd52e4e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c066830 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.855+0000 7fbd50bea700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c10fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.855+0000 7fbd50bea700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c10fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:53900/0 (socket says 172.21.15.181:53900) 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.855+0000 7fbd50bea700 1 -- 172.21.15.181:0/283743753 learned_addr learned my addr 172.21.15.181:0/283743753 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.855+0000 7fbd50bea700 1 -- 172.21.15.181:0/283743753 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40005040 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.856+0000 7fbd50bea700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c10fbe0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009680 tx=0x7fbd40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.856+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd4000bed0 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.856+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c066a30 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.856+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd4000b5c0 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.856+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd4001c5b0 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.856+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c110360 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.857+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fbd40013a80 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.857+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c105890 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.857+0000 7fbd497fa700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fbd3c036820 0x7fbd3c038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.857+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbd40049b70 con 0x7fbd4c0fbac0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.857+0000 7fbd4bfff700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fbd3c036820 0x7fbd3c038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:34.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.858+0000 7fbd4bfff700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fbd3c036820 0x7fbd3c038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009990 tx=0x7fbd38006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:34.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:34.862+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbd40011940 con 0x7fbd4c0fbac0 2024-02-10T06:35:35.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.117+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd4c04bed0 con 0x7fbd4c0fbac0 2024-02-10T06:35:35.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.117+0000 7fbd497fa700 1 -- 172.21.15.181:0/283743753 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbd40011370 con 0x7fbd4c0fbac0 2024-02-10T06:35:35.119 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:35.119 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:35.121 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.119+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fbd3c036820 msgr2=0x7fbd3c038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.119+0000 7fbd52e4e700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fbd3c036820 0x7fbd3c038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009990 tx=0x7fbd38006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.119+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 msgr2=0x7fbd4c10fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.119+0000 7fbd52e4e700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c10fbe0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009680 tx=0x7fbd40009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.120+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 shutdown_connections 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.120+0000 7fbd52e4e700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fbd3c036820 0x7fbd3c038cd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.120+0000 7fbd52e4e700 1 --2- 172.21.15.181:0/283743753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd4c0fbac0 0x7fbd4c10fbe0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.120+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 >> 172.21.15.181:0/283743753 conn(0x7fbd4c0f71a0 msgr2=0x7fbd4c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.120+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 shutdown_connections 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:35.120+0000 7fbd52e4e700 1 -- 172.21.15.181:0/283743753 wait complete. 2024-02-10T06:35:35.122 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:35.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:35 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/283743753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:36.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:36 smithi073 ceph-mon[29063]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:36.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:36.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:36.925 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:36.972 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:38.369 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:38 smithi073 ceph-mon[29063]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 -- 172.21.15.181:0/4018918098 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 msgr2=0x7feb000fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 --2- 172.21.15.181:0/4018918098 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fb290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feafc0048c0 tx=0x7feafc009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 -- 172.21.15.181:0/4018918098 shutdown_connections 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 --2- 172.21.15.181:0/4018918098 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fb290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 -- 172.21.15.181:0/4018918098 >> 172.21.15.181:0/4018918098 conn(0x7feb000f6920 msgr2=0x7feb000f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 -- 172.21.15.181:0/4018918098 shutdown_connections 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.463+0000 7feb074ba700 1 -- 172.21.15.181:0/4018918098 wait complete. 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb074ba700 1 Processor -- start 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb074ba700 1 -- start start 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb074ba700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:38.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb074ba700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb000fbe20 con 0x7feb000fae80 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb05256700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb05256700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:53920/0 (socket says 172.21.15.181:53920) 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb05256700 1 -- 172.21.15.181:0/2202250640 learned_addr learned my addr 172.21.15.181:0/2202250640 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb05256700 1 -- 172.21.15.181:0/2202250640 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc005040 con 0x7feb000fae80 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.464+0000 7feb05256700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fd9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feafc009040 tx=0x7feafc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.465+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feafc00bed0 con 0x7feb000fae80 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.465+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feafc00b5c0 con 0x7feb000fae80 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.465+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feafc01c5b0 con 0x7feb000fae80 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.465+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb000fc020 con 0x7feb000fae80 2024-02-10T06:35:38.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.465+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb000fc510 con 0x7feb000fae80 2024-02-10T06:35:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.466+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7feafc004d30 con 0x7feb000fae80 2024-02-10T06:35:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.466+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb00106930 con 0x7feb000fae80 2024-02-10T06:35:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.466+0000 7feaf5ffb700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7feaec0368c0 0x7feaec038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.466+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7feafc049b00 con 0x7feb000fae80 2024-02-10T06:35:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.466+0000 7feb04a55700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7feaec0368c0 0x7feaec038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:38.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.467+0000 7feb04a55700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7feaec0368c0 0x7feaec038d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009990 tx=0x7feaf0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:38.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.472+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7feafc011550 con 0x7feb000fae80 2024-02-10T06:35:38.730 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.729+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb0004bed0 con 0x7feb000fae80 2024-02-10T06:35:38.731 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.729+0000 7feaf5ffb700 1 -- 172.21.15.181:0/2202250640 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7feafc011370 con 0x7feb000fae80 2024-02-10T06:35:38.731 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:38.731 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:38.733 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7feaec0368c0 msgr2=0x7feaec038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:38.733 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7feaec0368c0 0x7feaec038d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009990 tx=0x7feaf0006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:38.733 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 msgr2=0x7feb000fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fd9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feafc009040 tx=0x7feafc009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 shutdown_connections 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7feaec0368c0 0x7feaec038d70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 --2- 172.21.15.181:0/2202250640 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb000fae80 0x7feb000fd9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 >> 172.21.15.181:0/2202250640 conn(0x7feb000f6920 msgr2=0x7feb000f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 shutdown_connections 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:38.731+0000 7feb074ba700 1 -- 172.21.15.181:0/2202250640 wait complete. 2024-02-10T06:35:38.734 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: Deploying daemon prometheus.smithi073 on smithi073 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/2202250640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:39.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:39 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:40.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:40.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:40.561 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:40.609 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 -- 172.21.15.181:0/1496806993 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 msgr2=0x7f9c7c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 --2- 172.21.15.181:0/1496806993 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c0fcfe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0048c0 tx=0x7f9c6c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 -- 172.21.15.181:0/1496806993 shutdown_connections 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 --2- 172.21.15.181:0/1496806993 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c0fcfe0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 -- 172.21.15.181:0/1496806993 >> 172.21.15.181:0/1496806993 conn(0x7f9c7c0f8290 msgr2=0x7f9c7c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 -- 172.21.15.181:0/1496806993 shutdown_connections 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.959+0000 7f9c820e2700 1 -- 172.21.15.181:0/1496806993 wait complete. 2024-02-10T06:35:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c820e2700 1 Processor -- start 2024-02-10T06:35:41.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c820e2700 1 -- start start 2024-02-10T06:35:41.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c820e2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:41.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c820e2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c10d400 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c7b7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:41.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c7b7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:53946/0 (socket says 172.21.15.181:53946) 2024-02-10T06:35:41.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.960+0000 7f9c7b7fe700 1 -- 172.21.15.181:0/1880044655 learned_addr learned my addr 172.21.15.181:0/1880044655 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:41.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c7b7fe700 1 -- 172.21.15.181:0/1880044655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c005040 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c7b7fe700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c10cec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009680 tx=0x7f9c6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:41.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c6c00bed0 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c10d600 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c6c00b5c0 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c6c01c5b0 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.961+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c10e2a0 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.962+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9c6c013420 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.962+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7c05f5c0 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.962+0000 7f9c73fff700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f9c68036820 0x7f9c68038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.962+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9c6c049c10 con 0x7f9c7c0fcbd0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.962+0000 7f9c7affd700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f9c68036820 0x7f9c68038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:41.964 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.963+0000 7f9c7affd700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f9c68036820 0x7f9c68038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009990 tx=0x7f9c64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:41.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:41.967+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9c6c013a80 con 0x7f9c7c0fcbd0 2024-02-10T06:35:42.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.224+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c7c10edf0 con 0x7f9c7c0fcbd0 2024-02-10T06:35:42.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.225+0000 7f9c73fff700 1 -- 172.21.15.181:0/1880044655 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9c6c011370 con 0x7f9c7c0fcbd0 2024-02-10T06:35:42.226 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:42.226 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f9c68036820 msgr2=0x7f9c68038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f9c68036820 0x7f9c68038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009990 tx=0x7f9c64006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 msgr2=0x7f9c7c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c10cec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009680 tx=0x7f9c6c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 shutdown_connections 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f9c68036820 0x7f9c68038cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:42.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 --2- 172.21.15.181:0/1880044655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c0fcbd0 0x7f9c7c10cec0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:42.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 >> 172.21.15.181:0/1880044655 conn(0x7f9c7c0f8290 msgr2=0x7f9c7c0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:42.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 shutdown_connections 2024-02-10T06:35:42.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:42.227+0000 7f9c820e2700 1 -- 172.21.15.181:0/1880044655 wait complete. 2024-02-10T06:35:42.229 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:42.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:42 smithi073 ceph-mon[29063]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:42.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:42 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1880044655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:43.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:43.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:44.001 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:44.048 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:44.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:44 smithi073 ceph-mon[29063]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 -- 172.21.15.181:0/1889863550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 msgr2=0x7f7b600fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 --2- 172.21.15.181:0/1889863550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b600fbee0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0048c0 tx=0x7f7b5c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 -- 172.21.15.181:0/1889863550 shutdown_connections 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 --2- 172.21.15.181:0/1889863550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b600fbee0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 -- 172.21.15.181:0/1889863550 >> 172.21.15.181:0/1889863550 conn(0x7f7b600f7190 msgr2=0x7f7b600f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 -- 172.21.15.181:0/1889863550 shutdown_connections 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 -- 172.21.15.181:0/1889863550 wait complete. 2024-02-10T06:35:45.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 Processor -- start 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.472+0000 7f7b67b71700 1 -- start start 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b67b71700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b60111fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b67b71700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6010d9c0 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b6590d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b60111fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b6590d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b60111fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:37206/0 (socket says 172.21.15.181:37206) 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b6590d700 1 -- 172.21.15.181:0/2492613517 learned_addr learned my addr 172.21.15.181:0/2492613517 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b6590d700 1 -- 172.21.15.181:0/2492613517 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c005040 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b6590d700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b60111fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009b80 tx=0x7f7b5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b5c00b5c0 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b5c00be70 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b60065d60 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b5c01c660 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.473+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b60112510 con 0x7f7b600fbad0 2024-02-10T06:35:45.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.474+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f7b5c01c8d0 con 0x7f7b600fbad0 2024-02-10T06:35:45.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.474+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b601058a0 con 0x7f7b600fbad0 2024-02-10T06:35:45.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.474+0000 7f7b567fc700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7b4c036870 0x7f7b4c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:45.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.474+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7b5c049bb0 con 0x7f7b600fbad0 2024-02-10T06:35:45.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.474+0000 7f7b6510c700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7b4c036870 0x7f7b4c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:45.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.475+0000 7f7b6510c700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7b4c036870 0x7f7b4c038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009990 tx=0x7f7b50006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:45.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.479+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7b5c011550 con 0x7f7b600fbad0 2024-02-10T06:35:45.740 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.739+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b6004bed0 con 0x7f7b600fbad0 2024-02-10T06:35:45.740 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.739+0000 7f7b567fc700 1 -- 172.21.15.181:0/2492613517 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7b5c004d30 con 0x7f7b600fbad0 2024-02-10T06:35:45.740 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:45.741 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:45.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7b4c036870 msgr2=0x7f7b4c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:45.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7b4c036870 0x7f7b4c038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009990 tx=0x7f7b50006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:45.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 msgr2=0x7f7b60111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:45.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b60111fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009b80 tx=0x7f7b5c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:45.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 shutdown_connections 2024-02-10T06:35:45.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7f7b4c036870 0x7f7b4c038d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:45.743 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 --2- 172.21.15.181:0/2492613517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b600fbad0 0x7f7b60111fd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:45.743 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 >> 172.21.15.181:0/2492613517 conn(0x7f7b600f7190 msgr2=0x7f7b600f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:45.743 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 shutdown_connections 2024-02-10T06:35:45.743 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:45.741+0000 7f7b67b71700 1 -- 172.21.15.181:0/2492613517 wait complete. 2024-02-10T06:35:45.743 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:46.529 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:46 smithi073 ceph-mon[29063]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:46.530 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:46 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/2492613517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:47.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:47.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:47.611 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:47.658 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:48.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:48 smithi073 ceph-mon[29063]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 -- 172.21.15.181:0/452051303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 msgr2=0x7fd4f00fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 --2- 172.21.15.181:0/452051303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f00fb2b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40048c0 tx=0x7fd4e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 -- 172.21.15.181:0/452051303 shutdown_connections 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 --2- 172.21.15.181:0/452051303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f00fb2b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 -- 172.21.15.181:0/452051303 >> 172.21.15.181:0/452051303 conn(0x7fd4f00f6920 msgr2=0x7fd4f00f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 -- 172.21.15.181:0/452051303 shutdown_connections 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.035+0000 7fd4f4879700 1 -- 172.21.15.181:0/452051303 wait complete. 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4f4879700 1 Processor -- start 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4f4879700 1 -- start start 2024-02-10T06:35:49.038 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4f4879700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4f4879700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f010d420 con 0x7fd4f00faea0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4ee59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4ee59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f010cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:37222/0 (socket says 172.21.15.181:37222) 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4ee59c700 1 -- 172.21.15.181:0/1875880830 learned_addr learned my addr 172.21.15.181:0/1875880830 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.036+0000 7fd4ee59c700 1 -- 172.21.15.181:0/1875880830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4005040 con 0x7fd4f00faea0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.037+0000 7fd4ee59c700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f010cee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009330 tx=0x7fd4e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.037+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e4004e60 con 0x7fd4f00faea0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.037+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4e40096b0 con 0x7fd4f00faea0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.037+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f010d620 con 0x7fd4f00faea0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.037+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e401c5b0 con 0x7fd4f00faea0 2024-02-10T06:35:49.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.037+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f010e2c0 con 0x7fd4f00faea0 2024-02-10T06:35:49.040 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.038+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fd4e401c820 con 0x7fd4f00faea0 2024-02-10T06:35:49.040 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.038+0000 7fd4deffd700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fd4d8036870 0x7fd4d8038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:49.040 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.038+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f005f5c0 con 0x7fd4f00faea0 2024-02-10T06:35:49.040 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.038+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd4e4049c40 con 0x7fd4f00faea0 2024-02-10T06:35:49.041 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.038+0000 7fd4edd9b700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fd4d8036870 0x7fd4d8038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:49.041 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.038+0000 7fd4edd9b700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fd4d8036870 0x7fd4d8038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009990 tx=0x7fd4e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:49.048 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.047+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd4e40119c0 con 0x7fd4f00faea0 2024-02-10T06:35:49.303 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.302+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4f005f5c0 con 0x7fd4f00faea0 2024-02-10T06:35:49.303 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.302+0000 7fd4deffd700 1 -- 172.21.15.181:0/1875880830 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd4e4023990 con 0x7fd4f00faea0 2024-02-10T06:35:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:49.303 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fd4d8036870 msgr2=0x7fd4d8038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fd4d8036870 0x7fd4d8038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009990 tx=0x7fd4e0006e40 comp rx=0 tx=0).stop 2024-02-10T06:35:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 msgr2=0x7fd4f010cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f010cee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009330 tx=0x7fd4e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 shutdown_connections 2024-02-10T06:35:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fd4d8036870 0x7fd4d8038d20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 --2- 172.21.15.181:0/1875880830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4f00faea0 0x7fd4f010cee0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 >> 172.21.15.181:0/1875880830 conn(0x7fd4f00f6920 msgr2=0x7fd4f00f76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 shutdown_connections 2024-02-10T06:35:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:49.304+0000 7fd4f4879700 1 -- 172.21.15.181:0/1875880830 wait complete. 2024-02-10T06:35:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:50.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:50 smithi073 ceph-mon[29063]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:35:50.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:50 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1875880830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:50.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:50 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:50.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:50 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:50.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:50 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:50.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:50 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T06:35:50.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:50.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:51.101 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:51.149 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:52.223 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:51 smithi073 ceph-mon[29063]: from='mgr.14162 172.21.15.73:0/4258157562' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T06:35:52.223 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:51 smithi073 ceph-mon[29063]: mgrmap e14: smithi073.vjivfn(active, since 62s) 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/4107762552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 msgr2=0x7fb6c80fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 --2- 172.21.15.181:0/4107762552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c80fbed0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80048c0 tx=0x7fb6b8009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/4107762552 shutdown_connections 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 --2- 172.21.15.181:0/4107762552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c80fbed0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/4107762552 >> 172.21.15.181:0/4107762552 conn(0x7fb6c80f71a0 msgr2=0x7fb6c80f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/4107762552 shutdown_connections 2024-02-10T06:35:52.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.548+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/4107762552 wait complete. 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6cc8d8700 1 Processor -- start 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6cc8d8700 1 -- start start 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6cc8d8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c810fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6cc8d8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8066810 con 0x7fb6c80fbac0 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6c659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c810fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6c659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c810fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:37234/0 (socket says 172.21.15.181:37234) 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.549+0000 7fb6c659c700 1 -- 172.21.15.181:0/3301515732 learned_addr learned my addr 172.21.15.181:0/3301515732 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6c659c700 1 -- 172.21.15.181:0/3301515732 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8005040 con 0x7fb6c80fbac0 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6c659c700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c810fbd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009680 tx=0x7fb6b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b800bed0 con 0x7fb6c80fbac0 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6b800b5c0 con 0x7fb6c80fbac0 2024-02-10T06:35:52.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c8066a10 con 0x7fb6c80fbac0 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b801c5b0 con 0x7fb6c80fbac0 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.550+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c8110350 con 0x7fb6c80fbac0 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.551+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7fb6b801c820 con 0x7fb6c80fbac0 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.551+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c8105890 con 0x7fb6c80fbac0 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.551+0000 7fb6b6ffd700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 0x7fb6b0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.551+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb6b8049ba0 con 0x7fb6c80fbac0 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.551+0000 7fb6c5d9b700 1 -- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 msgr2=0x7fb6b0038d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2308737934 2024-02-10T06:35:52.554 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.551+0000 7fb6c5d9b700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 0x7fb6b0038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:35:52.556 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.556+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6b8024cf0 con 0x7fb6c80fbac0 2024-02-10T06:35:52.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.752+0000 7fb6c5d9b700 1 -- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 msgr2=0x7fb6b0038d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2308737934 2024-02-10T06:35:52.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.752+0000 7fb6c5d9b700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 0x7fb6b0038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:35:52.814 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.813+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6c804bed0 con 0x7fb6c80fbac0 2024-02-10T06:35:52.815 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.813+0000 7fb6b6ffd700 1 -- 172.21.15.181:0/3301515732 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb6b8024b10 con 0x7fb6c80fbac0 2024-02-10T06:35:52.815 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:52.815 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:52.816 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 msgr2=0x7fb6b0038d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:35:52.816 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 0x7fb6b0038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 msgr2=0x7fb6c810fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c810fbd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009680 tx=0x7fb6b8009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 shutdown_connections 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fb6b0036870 0x7fb6b0038d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 --2- 172.21.15.181:0/3301515732 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb6c80fbac0 0x7fb6c810fbd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 >> 172.21.15.181:0/3301515732 conn(0x7fb6c80f71a0 msgr2=0x7fb6c80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 shutdown_connections 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:52.815+0000 7fb6cc8d8700 1 -- 172.21.15.181:0/3301515732 wait complete. 2024-02-10T06:35:52.817 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:53.140 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:52 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3301515732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:54.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:54.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:54.633 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:54.679 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:56.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 -- 172.21.15.181:0/1642806594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 msgr2=0x7fa2580fbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 --2- 172.21.15.181:0/1642806594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa2580fbef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0048c0 tx=0x7fa24c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 -- 172.21.15.181:0/1642806594 shutdown_connections 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 --2- 172.21.15.181:0/1642806594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa2580fbef0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 -- 172.21.15.181:0/1642806594 >> 172.21.15.181:0/1642806594 conn(0x7fa2580f71a0 msgr2=0x7fa2580f95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 -- 172.21.15.181:0/1642806594 shutdown_connections 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.144+0000 7fa25ca06700 1 -- 172.21.15.181:0/1642806594 wait complete. 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25ca06700 1 Processor -- start 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25ca06700 1 -- start start 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25ca06700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa25810cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25ca06700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25810d460 con 0x7fa2580fbae0 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa25810cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa25810cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:55660/0 (socket says 172.21.15.181:55660) 2024-02-10T06:35:56.147 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25659c700 1 -- 172.21.15.181:0/3069646000 learned_addr learned my addr 172.21.15.181:0/3069646000 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.145+0000 7fa25659c700 1 -- 172.21.15.181:0/3069646000 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c005040 con 0x7fa2580fbae0 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.146+0000 7fa25659c700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa25810cf20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009680 tx=0x7fa24c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.146+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24c00bed0 con 0x7fa2580fbae0 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.146+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa24c00b5c0 con 0x7fa2580fbae0 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.146+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25810d660 con 0x7fa2580fbae0 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.146+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24c01c5b0 con 0x7fa2580fbae0 2024-02-10T06:35:56.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.146+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25810e300 con 0x7fa2580fbae0 2024-02-10T06:35:56.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.147+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7fa24c013420 con 0x7fa2580fbae0 2024-02-10T06:35:56.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.147+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25805f5c0 con 0x7fa2580fbae0 2024-02-10T06:35:56.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.147+0000 7fa246ffd700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 0x7fa240038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:56.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.147+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa24c049ac0 con 0x7fa2580fbae0 2024-02-10T06:35:56.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.147+0000 7fa255d9b700 1 -- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 msgr2=0x7fa240038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2308737934 2024-02-10T06:35:56.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.147+0000 7fa255d9b700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 0x7fa240038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:35:56.152 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.152+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa24c04be60 con 0x7fa2580fbae0 2024-02-10T06:35:56.349 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.348+0000 7fa255d9b700 1 -- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 msgr2=0x7fa240038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2308737934 2024-02-10T06:35:56.349 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.348+0000 7fa255d9b700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 0x7fa240038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:35:56.409 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.408+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa25810ee50 con 0x7fa2580fbae0 2024-02-10T06:35:56.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.409+0000 7fa246ffd700 1 -- 172.21.15.181:0/3069646000 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa24c025400 con 0x7fa2580fbae0 2024-02-10T06:35:56.410 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:35:56.410 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 msgr2=0x7fa240038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 0x7fa240038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 msgr2=0x7fa25810cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa25810cf20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009680 tx=0x7fa24c009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 shutdown_connections 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:6800/2308737934,v1:172.21.15.73:6801/2308737934] conn(0x7fa2400368c0 0x7fa240038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 --2- 172.21.15.181:0/3069646000 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2580fbae0 0x7fa25810cf20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 >> 172.21.15.181:0/3069646000 conn(0x7fa2580f71a0 msgr2=0x7fa258066470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 shutdown_connections 2024-02-10T06:35:56.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:56.411+0000 7fa25ca06700 1 -- 172.21.15.181:0/3069646000 wait complete. 2024-02-10T06:35:56.413 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:35:57.483 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:57 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3069646000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:35:57.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:35:57.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:35:58.215 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:58.271 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:35:58.524 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: Active manager daemon smithi073.vjivfn restarted 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: Activating manager daemon smithi073.vjivfn 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: mgrmap e15: smithi073.vjivfn(active, starting, since 0.00584467s) 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr metadata", "who": "smithi073.vjivfn", "id": "smithi073.vjivfn"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: Manager daemon smithi073.vjivfn is now available 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:58.525 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:35:58.526 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.vjivfn/trash_purge_schedule"}]: dispatch 2024-02-10T06:35:58.526 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:59.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:59.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: mgrmap e16: smithi073.vjivfn(active, since 1.00802s) 2024-02-10T06:35:59.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: [10/Feb/2024:06:35:58] ENGINE Bus STARTING 2024-02-10T06:35:59.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: [10/Feb/2024:06:35:58] ENGINE Serving on https://172.21.15.73:7150 2024-02-10T06:35:59.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: [10/Feb/2024:06:35:58] ENGINE Bus STARTED 2024-02-10T06:35:59.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:59.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:35:59 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 -- 172.21.15.181:0/2720110344 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 msgr2=0x7f981c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 --2- 172.21.15.181:0/2720110344 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c0fcfe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f98100048c0 tx=0x7f9810009e30 comp rx=0 tx=0).stop 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 -- 172.21.15.181:0/2720110344 shutdown_connections 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 --2- 172.21.15.181:0/2720110344 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c0fcfe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 -- 172.21.15.181:0/2720110344 >> 172.21.15.181:0/2720110344 conn(0x7f981c0f8270 msgr2=0x7f981c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 -- 172.21.15.181:0/2720110344 shutdown_connections 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.792+0000 7f9822b56700 1 -- 172.21.15.181:0/2720110344 wait complete. 2024-02-10T06:35:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.793+0000 7f9822b56700 1 Processor -- start 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.793+0000 7f9822b56700 1 -- start start 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.793+0000 7f9822b56700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c10cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.793+0000 7f9822b56700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c10d4a0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.794+0000 7f98208f2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c10cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.794+0000 7f98208f2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c10cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:55666/0 (socket says 172.21.15.181:55666) 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.794+0000 7f98208f2700 1 -- 172.21.15.181:0/3376926900 learned_addr learned my addr 172.21.15.181:0/3376926900 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.794+0000 7f98208f2700 1 -- 172.21.15.181:0/3376926900 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810005040 con 0x7f981c0fcbd0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.794+0000 7f98208f2700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c10cf60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9810009680 tx=0x7f9810009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.794+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981000bed0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f981000b5c0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c10d6a0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981001c5b0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c10e2f0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.799 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f981001c820 con 0x7f981c0fcbd0 2024-02-10T06:35:59.799 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f98197fa700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f980c0366b0 0x7f980c038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:35:59.799 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.795+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f981004a9c0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.799 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.796+0000 7f981bfff700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f980c0366b0 0x7f980c038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:35:59.799 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.796+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c05f5c0 con 0x7f981c0fcbd0 2024-02-10T06:35:59.799 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.796+0000 7f981bfff700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f980c0366b0 0x7f980c038b60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9808009990 tx=0x7f9808006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:35:59.801 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:35:59.801+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9810023300 con 0x7f981c0fcbd0 2024-02-10T06:36:00.075 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.074+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f981c10ed00 con 0x7f981c0fcbd0 2024-02-10T06:36:00.076 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.075+0000 7f98197fa700 1 -- 172.21.15.181:0/3376926900 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9810048030 con 0x7f981c0fcbd0 2024-02-10T06:36:00.076 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:00.076 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f980c0366b0 msgr2=0x7f980c038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f980c0366b0 0x7f980c038b60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9808009990 tx=0x7f9808006e40 comp rx=0 tx=0).stop 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 msgr2=0x7f981c10cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c10cf60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9810009680 tx=0x7f9810009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 shutdown_connections 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f980c0366b0 0x7f980c038b60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 --2- 172.21.15.181:0/3376926900 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f981c0fcbd0 0x7f981c10cf60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:00.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 >> 172.21.15.181:0/3376926900 conn(0x7f981c0f8270 msgr2=0x7f981c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:00.079 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 shutdown_connections 2024-02-10T06:36:00.079 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:00.077+0000 7f9822b56700 1 -- 172.21.15.181:0/3376926900 wait complete. 2024-02-10T06:36:00.079 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:36:00.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:00 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3376926900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:01.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:01 smithi073 ceph-mon[29063]: mgrmap e17: smithi073.vjivfn(active, since 2s) 2024-02-10T06:36:02.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:36:02.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:36:03.162 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:36:03.210 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:36:04.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:04.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:04.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:04.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:04.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:36:04.662 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.659+0000 7ff0e28ca700 1 -- 172.21.15.181:0/2660152372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 msgr2=0x7ff0dc0fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:04.662 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.659+0000 7ff0e28ca700 1 --2- 172.21.15.181:0/2660152372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc0fcfd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00048c0 tx=0x7ff0d0009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.659+0000 7ff0e28ca700 1 -- 172.21.15.181:0/2660152372 shutdown_connections 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.659+0000 7ff0e28ca700 1 --2- 172.21.15.181:0/2660152372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc0fcfd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.659+0000 7ff0e28ca700 1 -- 172.21.15.181:0/2660152372 >> 172.21.15.181:0/2660152372 conn(0x7ff0dc0f82a0 msgr2=0x7ff0dc0fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0e28ca700 1 -- 172.21.15.181:0/2660152372 shutdown_connections 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0e28ca700 1 -- 172.21.15.181:0/2660152372 wait complete. 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0e28ca700 1 Processor -- start 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0e28ca700 1 -- start start 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0e28ca700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0e28ca700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc10d3d0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0dbfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0dbfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:35698/0 (socket says 172.21.15.181:35698) 2024-02-10T06:36:04.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.660+0000 7ff0dbfff700 1 -- 172.21.15.181:0/1190709631 learned_addr learned my addr 172.21.15.181:0/1190709631 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0dbfff700 1 -- 172.21.15.181:0/1190709631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d0005040 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0dbfff700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc10ce90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009680 tx=0x7ff0d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0d000bed0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc10d5d0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0d000b5c0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0d001c5b0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.664 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.661+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc10e270 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.662+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0dc05f5c0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.662+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7ff0d0013a80 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.662+0000 7ff0d8ff9700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff0c4036910 0x7ff0c4038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:04.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.662+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff0d0049cc0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.662+0000 7ff0db7fe700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff0c4036910 0x7ff0c4038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:04.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.663+0000 7ff0db7fe700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff0c4036910 0x7ff0c4038dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009990 tx=0x7ff0cc006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:04.668 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.667+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff0d0011940 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.921 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.921+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0dc05f5c0 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.922 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.921+0000 7ff0d8ff9700 1 -- 172.21.15.181:0/1190709631 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff0d0011370 con 0x7ff0dc0fcbc0 2024-02-10T06:36:04.922 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:04.922 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff0c4036910 msgr2=0x7ff0c4038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff0c4036910 0x7ff0c4038dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009990 tx=0x7ff0cc006e40 comp rx=0 tx=0).stop 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 msgr2=0x7ff0dc10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc10ce90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009680 tx=0x7ff0d0009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 shutdown_connections 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff0c4036910 0x7ff0c4038dc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 --2- 172.21.15.181:0/1190709631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0dc0fcbc0 0x7ff0dc10ce90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:04.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 >> 172.21.15.181:0/1190709631 conn(0x7ff0dc0f82a0 msgr2=0x7ff0dc0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:04.925 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 shutdown_connections 2024-02-10T06:36:04.925 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:04.923+0000 7ff0e28ca700 1 -- 172.21.15.181:0/1190709631 wait complete. 2024-02-10T06:36:04.925 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:36:05.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:04 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1190709631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:06.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:36:06.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:36:06.737 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:36:06.784 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.231+0000 7f79262a0700 1 -- 172.21.15.181:0/172014576 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 msgr2=0x7f79200fe630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.231+0000 7f79262a0700 1 --2- 172.21.15.181:0/172014576 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f79200fe630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f79100048c0 tx=0x7f7910009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.231+0000 7f79262a0700 1 -- 172.21.15.181:0/172014576 shutdown_connections 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.231+0000 7f79262a0700 1 --2- 172.21.15.181:0/172014576 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f79200fe630 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.231+0000 7f79262a0700 1 -- 172.21.15.181:0/172014576 >> 172.21.15.181:0/172014576 conn(0x7f79200f6920 msgr2=0x7f79200f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.232+0000 7f79262a0700 1 -- 172.21.15.181:0/172014576 shutdown_connections 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.232+0000 7f79262a0700 1 -- 172.21.15.181:0/172014576 wait complete. 2024-02-10T06:36:08.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.232+0000 7f79262a0700 1 Processor -- start 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.232+0000 7f79262a0700 1 -- start start 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.232+0000 7f79262a0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f792010cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.232+0000 7f79262a0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792010d440 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f792010cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f792010cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:35708/0 (socket says 172.21.15.181:35708) 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791ffff700 1 -- 172.21.15.181:0/3360100690 learned_addr learned my addr 172.21.15.181:0/3360100690 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791ffff700 1 -- 172.21.15.181:0/3360100690 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7910005040 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791ffff700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f792010cf00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7910009680 tx=0x7f7910009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791000bed0 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f791000b5c0 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792010d640 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791001c5b0 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.233+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792010e2e0 con 0x7f79200fe220 2024-02-10T06:36:08.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.234+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f791001c820 con 0x7f79200fe220 2024-02-10T06:36:08.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.234+0000 7f791cff9700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7908036910 0x7f7908038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:08.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.234+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7910049d10 con 0x7f79200fe220 2024-02-10T06:36:08.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.235+0000 7f791f7fe700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7908036910 0x7f7908038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:08.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.235+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792005f5c0 con 0x7f79200fe220 2024-02-10T06:36:08.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.235+0000 7f791f7fe700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7908036910 0x7f7908038dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7914009990 tx=0x7f7914006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:08.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.239+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7910023300 con 0x7f79200fe220 2024-02-10T06:36:08.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.510+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f792010ecf0 con 0x7f79200fe220 2024-02-10T06:36:08.512 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.511+0000 7f791cff9700 1 -- 172.21.15.181:0/3360100690 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7910011b60 con 0x7f79200fe220 2024-02-10T06:36:08.512 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:08.512 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:36:08.514 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7908036910 msgr2=0x7f7908038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:08.514 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7908036910 0x7f7908038dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7914009990 tx=0x7f7914006e40 comp rx=0 tx=0).stop 2024-02-10T06:36:08.514 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 msgr2=0x7f792010cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:08.514 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f792010cf00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7910009680 tx=0x7f7910009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 shutdown_connections 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7908036910 0x7f7908038dc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 --2- 172.21.15.181:0/3360100690 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f79200fe220 0x7f792010cf00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 >> 172.21.15.181:0/3360100690 conn(0x7f79200f6920 msgr2=0x7f79200f76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 shutdown_connections 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:08.513+0000 7f79262a0700 1 -- 172.21.15.181:0/3360100690 wait complete. 2024-02-10T06:36:08.515 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:08.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:36:09.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi073:/etc/ceph/ceph.conf 2024-02-10T06:36:09.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi181:/etc/ceph/ceph.conf 2024-02-10T06:36:09.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi073:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:09.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi181:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:09.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3360100690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi073:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Updating smithi181:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi181", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi181", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:09.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:09 smithi073 ceph-mon[29063]: Deploying daemon crash.smithi181 on smithi181 2024-02-10T06:36:10.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:36:10.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:36:10.455 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 -- 172.21.15.181:0/2215143540 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 msgr2=0x7f91780fe020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 --2- 172.21.15.181:0/2215143540 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f91780fe020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9170009c60 tx=0x7f9170004d10 comp rx=0 tx=0).stop 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 -- 172.21.15.181:0/2215143540 shutdown_connections 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 --2- 172.21.15.181:0/2215143540 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f91780fe020 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 -- 172.21.15.181:0/2215143540 >> 172.21.15.181:0/2215143540 conn(0x7f9178072850 msgr2=0x7f9178072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 -- 172.21.15.181:0/2215143540 shutdown_connections 2024-02-10T06:36:12.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.363+0000 7f917c9f3700 1 -- 172.21.15.181:0/2215143540 wait complete. 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917c9f3700 1 Processor -- start 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917c9f3700 1 -- start start 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917c9f3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f9178112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917c9f3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9178113500 con 0x7f91780fdc50 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f9178112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f9178112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:35724/0 (socket says 172.21.15.181:35724) 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.364+0000 7f917659c700 1 -- 172.21.15.181:0/2014957994 learned_addr learned my addr 172.21.15.181:0/2014957994 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f917659c700 1 -- 172.21.15.181:0/2014957994 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170009940 con 0x7f91780fdc50 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f917659c700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f9178112fc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9170004e60 tx=0x7f9170004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9170007c80 con 0x7f91780fdc50 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9178113700 con 0x7f91780fdc50 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f917001d910 con 0x7f91780fdc50 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9178113b70 con 0x7f91780fdc50 2024-02-10T06:36:12.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.365+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91700129f0 con 0x7f91780fdc50 2024-02-10T06:36:12.368 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.366+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917805f5c0 con 0x7f91780fdc50 2024-02-10T06:36:12.368 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.366+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9170019040 con 0x7f91780fdc50 2024-02-10T06:36:12.368 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.366+0000 7f9166ffd700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9160036960 0x7f9160038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:12.369 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.366+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f917004a450 con 0x7f91780fdc50 2024-02-10T06:36:12.369 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.366+0000 7f9175d9b700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9160036960 0x7f9160038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:12.369 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.367+0000 7f9175d9b700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9160036960 0x7f9160038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f916c009990 tx=0x7f916c006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:12.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.371+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f917002eb40 con 0x7f91780fdc50 2024-02-10T06:36:12.638 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.637+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f917810e680 con 0x7f91780fdc50 2024-02-10T06:36:12.639 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.637+0000 7f9166ffd700 1 -- 172.21.15.181:0/2014957994 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9170017020 con 0x7f91780fdc50 2024-02-10T06:36:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:12.639 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:36:12.640 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9160036960 msgr2=0x7f9160038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9160036960 0x7f9160038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f916c009990 tx=0x7f916c006e40 comp rx=0 tx=0).stop 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 msgr2=0x7f9178112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f9178112fc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9170004e60 tx=0x7f9170004d10 comp rx=0 tx=0).stop 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 shutdown_connections 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9160036960 0x7f9160038e10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 --2- 172.21.15.181:0/2014957994 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91780fdc50 0x7f9178112fc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 >> 172.21.15.181:0/2014957994 conn(0x7f9178072850 msgr2=0x7f9178101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 shutdown_connections 2024-02-10T06:36:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:12.640+0000 7f917c9f3700 1 -- 172.21.15.181:0/2014957994 wait complete. 2024-02-10T06:36:12.642 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:36:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:13.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:13 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/2014957994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:13.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:13.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:13.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:13.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:13.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:13 smithi073 ceph-mon[29063]: Deploying daemon node-exporter.smithi181 on smithi181 2024-02-10T06:36:14.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:36:14.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:36:14.576 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:17.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 -- 172.21.15.181:0/2902839157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 msgr2=0x7f036c101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:17.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 --2- 172.21.15.181:0/2902839157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c101f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f03600048c0 tx=0x7f0360009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:17.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 -- 172.21.15.181:0/2902839157 shutdown_connections 2024-02-10T06:36:17.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 --2- 172.21.15.181:0/2902839157 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c101f30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:17.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 -- 172.21.15.181:0/2902839157 >> 172.21.15.181:0/2902839157 conn(0x7f036c072850 msgr2=0x7f036c072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 -- 172.21.15.181:0/2902839157 shutdown_connections 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.667+0000 7f037284f700 1 -- 172.21.15.181:0/2902839157 wait complete. 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.668+0000 7f037284f700 1 Processor -- start 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.668+0000 7f037284f700 1 -- start start 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.668+0000 7f037284f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.668+0000 7f037284f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1133f0 con 0x7f036c101b60 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f036bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f036bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c112eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:53292/0 (socket says 172.21.15.181:53292) 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f036bfff700 1 -- 172.21.15.181:0/2611568587 learned_addr learned my addr 172.21.15.181:0/2611568587 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f036bfff700 1 -- 172.21.15.181:0/2611568587 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360005040 con 0x7f036c101b60 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f036bfff700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c112eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0360009300 tx=0x7f0360009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:17.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360004e60 con 0x7f036c101b60 2024-02-10T06:36:17.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c1135f0 con 0x7f036c101b60 2024-02-10T06:36:17.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0360013420 con 0x7f036c101b60 2024-02-10T06:36:17.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c113ae0 con 0x7f036c101b60 2024-02-10T06:36:17.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.669+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036001c590 con 0x7f036c101b60 2024-02-10T06:36:17.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.670+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f036001c850 con 0x7f036c101b60 2024-02-10T06:36:17.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.670+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c05f5c0 con 0x7f036c101b60 2024-02-10T06:36:17.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.670+0000 7f0368ff9700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0354036910 0x7f0354038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:17.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.670+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0360049d50 con 0x7f036c101b60 2024-02-10T06:36:17.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.671+0000 7f036b7fe700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0354036910 0x7f0354038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:17.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.671+0000 7f036b7fe700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0354036910 0x7f0354038dc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f035c009990 tx=0x7f035c006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:17.676 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.676+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0360009600 con 0x7f036c101b60 2024-02-10T06:36:17.947 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.946+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f036c105dc0 con 0x7f036c101b60 2024-02-10T06:36:17.948 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.947+0000 7f0368ff9700 1 -- 172.21.15.181:0/2611568587 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0360009600 con 0x7f036c101b60 2024-02-10T06:36:17.948 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:17.948 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:36:17.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0354036910 msgr2=0x7f0354038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:17.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0354036910 0x7f0354038dc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f035c009990 tx=0x7f035c006e40 comp rx=0 tx=0).stop 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 msgr2=0x7f036c112eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c112eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0360009300 tx=0x7f0360009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 shutdown_connections 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0354036910 0x7f0354038dc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 --2- 172.21.15.181:0/2611568587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f036c101b60 0x7f036c112eb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 >> 172.21.15.181:0/2611568587 conn(0x7f036c072850 msgr2=0x7f036c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.949+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 shutdown_connections 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:17.950+0000 7f037284f700 1 -- 172.21.15.181:0/2611568587 wait complete. 2024-02-10T06:36:17.951 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:36:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi181.lmidhc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi181.lmidhc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: Deploying daemon mgr.smithi181.lmidhc on smithi181 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:18.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:18 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/2611568587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:20.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:36:20.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:36:20.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:20.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:20.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:20.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:20.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:20.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:36:20.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:21.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:21 smithi073 ceph-mon[29063]: Deploying daemon mon.smithi181 on smithi181 2024-02-10T06:36:22.079 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:22.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:22 smithi073 ceph-mon[29063]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:23.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:23 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:24.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 -- 172.21.15.181:0/3717514068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 msgr2=0x7fc730102fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:24.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 --2- 172.21.15.181:0/3717514068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730102fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc7240048c0 tx=0x7fc724009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:24.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 -- 172.21.15.181:0/3717514068 shutdown_connections 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 --2- 172.21.15.181:0/3717514068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730102fb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 -- 172.21.15.181:0/3717514068 >> 172.21.15.181:0/3717514068 conn(0x7fc7300720b0 msgr2=0x7fc7300724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 -- 172.21.15.181:0/3717514068 shutdown_connections 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 -- 172.21.15.181:0/3717514068 wait complete. 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 Processor -- start 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.227+0000 7fc736b77700 1 -- start start 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc736b77700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc736b77700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc730113570 con 0x7fc730102be0 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc734913700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc734913700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.181:37434/0 (socket says 172.21.15.181:37434) 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc734913700 1 -- 172.21.15.181:0/1774833836 learned_addr learned my addr 172.21.15.181:0/1774833836 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc734913700 1 -- 172.21.15.181:0/1774833836 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724005040 con 0x7fc730102be0 2024-02-10T06:36:24.230 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc734913700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730113030 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc724009f80 tx=0x7fc724009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:24.231 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc724004e60 con 0x7fc730102be0 2024-02-10T06:36:24.231 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc730113770 con 0x7fc730102be0 2024-02-10T06:36:24.231 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc724013420 con 0x7fc730102be0 2024-02-10T06:36:24.231 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.228+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc72401c5b0 con 0x7fc730102be0 2024-02-10T06:36:24.231 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.229+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc730113be0 con 0x7fc730102be0 2024-02-10T06:36:24.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.229+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73005f5c0 con 0x7fc730102be0 2024-02-10T06:36:24.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.229+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fc724026440 con 0x7fc730102be0 2024-02-10T06:36:24.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.230+0000 7fc72d7fa700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc720036960 0x7fc720038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:24.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.230+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc724049610 con 0x7fc730102be0 2024-02-10T06:36:24.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.230+0000 7fc72ffff700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc720036960 0x7fc720038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:24.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.230+0000 7fc72ffff700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc720036960 0x7fc720038e10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009990 tx=0x7fc71c006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:24.236 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.235+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc724011780 con 0x7fc730102be0 2024-02-10T06:36:24.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.506+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc730106f20 con 0x7fc730102be0 2024-02-10T06:36:24.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.507+0000 7fc72d7fa700 1 -- 172.21.15.181:0/1774833836 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc724021030 con 0x7fc730102be0 2024-02-10T06:36:24.508 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:24.508 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:33:46.641473Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:36:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc720036960 msgr2=0x7fc720038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc720036960 0x7fc720038e10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009990 tx=0x7fc71c006e40 comp rx=0 tx=0).stop 2024-02-10T06:36:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 msgr2=0x7fc730113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730113030 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc724009f80 tx=0x7fc724009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 shutdown_connections 2024-02-10T06:36:24.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc720036960 0x7fc720038e10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:24.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 --2- 172.21.15.181:0/1774833836 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc730102be0 0x7fc730113030 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:24.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 >> 172.21.15.181:0/1774833836 conn(0x7fc7300720b0 msgr2=0x7fc7300f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:24.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 shutdown_connections 2024-02-10T06:36:24.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:24.509+0000 7fc736b77700 1 -- 172.21.15.181:0/1774833836 wait complete. 2024-02-10T06:36:24.511 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-10T06:36:24.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:24 smithi073 ceph-mon[29063]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:26.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:36:26.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:36:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:36:29.893 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:29 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:30.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:36:30.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: mon.smithi073 calling monitor election 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: mon.smithi181 calling monitor election 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.? 172.21.15.181:0/4010833301' entity='mgr.smithi181.lmidhc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi181.lmidhc/crt"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: mon.smithi073 is new leader, mons smithi073,smithi181 in quorum (ranks 0,1) 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: monmap e2: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi181=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: fsmap 2024-02-10T06:36:30.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: mgrmap e17: smithi073.vjivfn(active, since 32s) 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: Standby manager daemon smithi181.lmidhc started 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.? 172.21.15.181:0/4010833301' entity='mgr.smithi181.lmidhc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: overall HEALTH_OK 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.? 172.21.15.181:0/4010833301' entity='mgr.smithi181.lmidhc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi181.lmidhc/key"}]: dispatch 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.? 172.21.15.181:0/4010833301' entity='mgr.smithi181.lmidhc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:30.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:30.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.881+0000 7f26e0bb2700 1 -- 172.21.15.181:0/523216162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26c4003580 msgr2=0x7f26c4005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:30.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.881+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/523216162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26c4003580 0x7f26c4005a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f26dc065630 tx=0x7f26cc00b040 comp rx=0 tx=0).stop 2024-02-10T06:36:30.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.882+0000 7f26e0bb2700 1 -- 172.21.15.181:0/523216162 shutdown_connections 2024-02-10T06:36:30.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.882+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/523216162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26c4003580 0x7f26c4005a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:30.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.882+0000 7f26e0bb2700 1 -- 172.21.15.181:0/523216162 >> 172.21.15.181:0/523216162 conn(0x7f26dc0729f0 msgr2=0x7f26dc072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:30.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.882+0000 7f26e0bb2700 1 -- 172.21.15.181:0/523216162 shutdown_connections 2024-02-10T06:36:30.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.882+0000 7f26e0bb2700 1 -- 172.21.15.181:0/523216162 wait complete. 2024-02-10T06:36:30.885 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:30 smithi181 ceph-mon[36721]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:30.885 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:30 smithi181 ceph-mon[36721]: mgrmap e18: smithi073.vjivfn(active, since 32s), standbys: smithi181.lmidhc 2024-02-10T06:36:30.885 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr metadata", "who": "smithi181.lmidhc", "id": "smithi181.lmidhc"}]: dispatch 2024-02-10T06:36:30.885 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:30.885 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:36:30.885 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.882+0000 7f26e0bb2700 1 Processor -- start 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26e0bb2700 1 -- start start 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26e0bb2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26e0bb2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc113710 0x7f26dc10d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26e0bb2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc113d20 con 0x7f26dc113710 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26e0bb2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc10d5b0 con 0x7f26c4003580 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:33424/0 (socket says 172.21.15.181:33424) 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 -- 172.21.15.181:0/1809876697 learned_addr learned my addr 172.21.15.181:0/1809876697 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 -- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 msgr2=0x7f26dc1131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 -- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 msgr2=0x7f26dc1131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-10T06:36:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26d9d9b700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc113710 0x7f26dc10d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26da59c700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26d9d9b700 1 -- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 msgr2=0x7f26dc1131d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26d9d9b700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.883+0000 7f26d9d9b700 1 -- 172.21.15.181:0/1809876697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc005040 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.884+0000 7f26d9d9b700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc113710 0x7f26dc10d070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b790 tx=0x7f26d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.884+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d0011810 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.884+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d0011e40 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.884+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc10d890 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.884+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d000f2e0 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.884+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc10ddd0 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.885+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc05f5c0 con 0x7f26dc113710 2024-02-10T06:36:30.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.885+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f26d0010380 con 0x7f26dc113710 2024-02-10T06:36:30.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.886+0000 7f26caffd700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26bc069320 0x7f26bc06b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:30.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.886+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f26d0086360 con 0x7f26dc113710 2024-02-10T06:36:30.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.886+0000 7f26da59c700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26bc069320 0x7f26bc06b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:30.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.886+0000 7f26da59c700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26bc069320 0x7f26bc06b7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f26cc000ea0 tx=0x7f26cc018040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:30.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:30.891+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f26d0057180 con 0x7f26dc113710 2024-02-10T06:36:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:30 smithi073 ceph-mon[29063]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:30 smithi073 ceph-mon[29063]: mgrmap e18: smithi073.vjivfn(active, since 32s), standbys: smithi181.lmidhc 2024-02-10T06:36:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr metadata", "who": "smithi181.lmidhc", "id": "smithi181.lmidhc"}]: dispatch 2024-02-10T06:36:31.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:31.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:36:31.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:36:31.167 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.165+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26dc104a00 con 0x7f26dc113710 2024-02-10T06:36:31.168 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.166+0000 7f26caffd700 1 -- 172.21.15.181:0/1809876697 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f26d0015090 con 0x7f26dc113710 2024-02-10T06:36:31.168 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-10T06:36:31.168 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:24.528766Z","created":"2024-02-10T06:33:46.641473Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi181","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T06:36:31.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26bc069320 msgr2=0x7f26bc06b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:31.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26bc069320 0x7f26bc06b7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f26cc000ea0 tx=0x7f26cc018040 comp rx=0 tx=0).stop 2024-02-10T06:36:31.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc113710 msgr2=0x7f26dc10d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:31.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc113710 0x7f26dc10d070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b790 tx=0x7f26d00091a0 comp rx=0 tx=0).stop 2024-02-10T06:36:31.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 shutdown_connections 2024-02-10T06:36:31.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26bc069320 0x7f26bc06b7d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26c4003580 0x7f26dc1131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 --2- 172.21.15.181:0/1809876697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc113710 0x7f26dc10d070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 >> 172.21.15.181:0/1809876697 conn(0x7f26dc0729f0 msgr2=0x7f26dc0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 shutdown_connections 2024-02-10T06:36:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:36:31.168+0000 7f26e0bb2700 1 -- 172.21.15.181:0/1809876697 wait complete. 2024-02-10T06:36:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-02-10T06:36:31.860 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:31 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/1809876697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:31.861 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T06:36:31.862 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T06:36:31.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:31 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1809876697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:36:32.130 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:36:32.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:32.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:32.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:36:32.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:33.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:36:33.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.093 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:33.093 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:36:33.093 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:36:33.093 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:33.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:33 smithi073 ceph-mon[29063]: Updating smithi073:/etc/ceph/ceph.conf 2024-02-10T06:36:33.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:33 smithi073 ceph-mon[29063]: Updating smithi181:/etc/ceph/ceph.conf 2024-02-10T06:36:33.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:33 smithi073 ceph-mon[29063]: Updating smithi181:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:33.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:33 smithi073 ceph-mon[29063]: Updating smithi073:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:33.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:33 smithi073 ceph-mon[29063]: Reconfiguring mon.smithi073 (unknown last config time)... 2024-02-10T06:36:33.916 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:33 smithi073 ceph-mon[29063]: Reconfiguring daemon mon.smithi073 on smithi073 2024-02-10T06:36:34.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:33 smithi181 ceph-mon[36721]: Updating smithi073:/etc/ceph/ceph.conf 2024-02-10T06:36:34.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:33 smithi181 ceph-mon[36721]: Updating smithi181:/etc/ceph/ceph.conf 2024-02-10T06:36:34.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:33 smithi181 ceph-mon[36721]: Updating smithi181:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:34.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:33 smithi181 ceph-mon[36721]: Updating smithi073:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:36:34.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:33 smithi181 ceph-mon[36721]: Reconfiguring mon.smithi073 (unknown last config time)... 2024-02-10T06:36:34.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:33 smithi181 ceph-mon[36721]: Reconfiguring daemon mon.smithi073 on smithi073 2024-02-10T06:36:34.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.381+0000 7fa2b929b700 1 -- 172.21.15.73:0/1181814437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4071f60 msgr2=0x7fa2b4070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.381+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1181814437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4071f60 0x7fa2b4070330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40048c0 tx=0x7fa2a4009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.381+0000 7fa2b929b700 1 -- 172.21.15.73:0/1181814437 shutdown_connections 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.381+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1181814437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4071f60 0x7fa2b4070330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.381+0000 7fa2b929b700 1 -- 172.21.15.73:0/1181814437 >> 172.21.15.73:0/1181814437 conn(0x7fa2b40f8760 msgr2=0x7fa2b40fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.381+0000 7fa2b929b700 1 -- 172.21.15.73:0/1181814437 shutdown_connections 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 -- 172.21.15.73:0/1181814437 wait complete. 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 Processor -- start 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 -- start start 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2b4071f60 0x7fa2b4119840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 0x7fa2b4113700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b411a340 con 0x7fa2b4119d80 2024-02-10T06:36:34.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.382+0000 7fa2b929b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b4113c40 con 0x7fa2b4071f60 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 0x7fa2b4113700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 0x7fa2b4113700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:48740/0 (socket says 172.21.15.73:48740) 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 -- 172.21.15.73:0/1735884088 learned_addr learned my addr 172.21.15.73:0/1735884088 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 -- 172.21.15.73:0/1735884088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2b4071f60 msgr2=0x7fa2b4119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2b4071f60 0x7fa2b4119840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 -- 172.21.15.73:0/1735884088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4005040 con 0x7fa2b4119d80 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b27fc700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 0x7fa2b4113700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800b790 tx=0x7fa2a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2a8010870 con 0x7fa2b4119d80 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b4113f20 con 0x7fa2b4119d80 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2a800f3e0 con 0x7fa2b4119d80 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2a8010ea0 con 0x7fa2b4119d80 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.383+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b4114460 con 0x7fa2b4119d80 2024-02-10T06:36:34.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.384+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b405f5c0 con 0x7fa2b4119d80 2024-02-10T06:36:34.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.384+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fa2a800f550 con 0x7fa2b4119d80 2024-02-10T06:36:34.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.385+0000 7fa29b7fe700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa29c069370 0x7fa29c06b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:34.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.385+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa2a80866e0 con 0x7fa2b4119d80 2024-02-10T06:36:34.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.385+0000 7fa2b2ffd700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa29c069370 0x7fa29c06b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:34.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.385+0000 7fa2b2ffd700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa29c069370 0x7fa29c06b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009b80 tx=0x7fa2a4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:34.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.389+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa2a8057500 con 0x7fa2b4119d80 2024-02-10T06:36:34.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.581+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa2b404bed0 con 0x7fa2b4119d80 2024-02-10T06:36:34.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.582+0000 7fa29b7fe700 1 -- 172.21.15.73:0/1735884088 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fa2a8014100 con 0x7fa2b4119d80 2024-02-10T06:36:34.583 INFO:teuthology.orchestra.run.smithi073.stdout:# minimal ceph.conf for 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:34.583 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2024-02-10T06:36:34.583 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 3c2b87a2-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:34.583 INFO:teuthology.orchestra.run.smithi073.stdout: mon_host = [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa29c069370 msgr2=0x7fa29c06b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa29c069370 0x7fa29c06b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009b80 tx=0x7fa2a4009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 msgr2=0x7fa2b4113700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 0x7fa2b4113700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800b790 tx=0x7fa2a8009250 comp rx=0 tx=0).stop 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 shutdown_connections 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa29c069370 0x7fa29c06b820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2b4071f60 0x7fa2b4119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 --2- 172.21.15.73:0/1735884088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2b4119d80 0x7fa2b4113700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 >> 172.21.15.73:0/1735884088 conn(0x7fa2b40f8760 msgr2=0x7fa2b40f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:34.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 shutdown_connections 2024-02-10T06:36:34.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:34.584+0000 7fa2b929b700 1 -- 172.21.15.73:0/1735884088 wait complete. 2024-02-10T06:36:34.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:34 smithi073 ceph-mon[29063]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:34.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:34 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1735884088' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:35.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:34 smithi181 ceph-mon[36721]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:35.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:34 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1735884088' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:35.583 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T06:36:35.583 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:36:35.583 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T06:36:35.653 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:36:35.653 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:35.720 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:36:35.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T06:36:35.755 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:36:35.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:35.827 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T06:36:35.827 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:36:35.828 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T06:36:35.843 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T06:36:35.843 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-02-10 06:35:49.179826845 +0000 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-02-10 06:30:25.102365667 +0000 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-02-10 06:30:25.102365667 +0000 2024-02-10T06:36:35.900 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-02-10 06:30:25.102365667 +0000 2024-02-10T06:36:35.900 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T06:36:35.965 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-02-10T06:36:35.966 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-02-10T06:36:35.966 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000212749 s, 2.4 MB/s 2024-02-10T06:36:35.967 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T06:36:36.023 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-02-10 06:35:49.179826845 +0000 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-02-10 06:30:25.444357185 +0000 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-02-10 06:30:25.444357185 +0000 2024-02-10T06:36:36.078 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-02-10 06:30:25.444357185 +0000 2024-02-10T06:36:36.079 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T06:36:36.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:35 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:36.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:35 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:36.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:35 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.vjivfn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:36:36.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:35 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:36:36.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:35 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:36.142 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:35 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:36.142 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:35 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:36.142 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:35 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.vjivfn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:36:36.142 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:35 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:36:36.142 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:35 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:36.144 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-02-10T06:36:36.144 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-02-10T06:36:36.145 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000189674 s, 2.7 MB/s 2024-02-10T06:36:36.146 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T06:36:36.202 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2024-02-10T06:36:36.257 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T06:36:36.257 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:36.257 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T06:36:36.257 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:36.257 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:36.257 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-02-10 06:35:49.180826820 +0000 2024-02-10T06:36:36.258 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-02-10 06:30:25.783348775 +0000 2024-02-10T06:36:36.258 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-02-10 06:30:25.783348775 +0000 2024-02-10T06:36:36.258 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-02-10 06:30:25.783348775 +0000 2024-02-10T06:36:36.258 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T06:36:36.323 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-02-10T06:36:36.323 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-02-10T06:36:36.323 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.00019388 s, 2.6 MB/s 2024-02-10T06:36:36.324 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T06:36:36.380 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-02-10 06:35:49.180826820 +0000 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-02-10 06:30:26.131340142 +0000 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-02-10 06:30:26.131340142 +0000 2024-02-10T06:36:36.435 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-02-10 06:30:26.131340142 +0000 2024-02-10T06:36:36.435 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T06:36:36.501 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-02-10T06:36:36.501 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-02-10T06:36:36.501 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000188152 s, 2.7 MB/s 2024-02-10T06:36:36.502 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T06:36:36.558 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:36:36.558 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T06:36:36.576 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T06:36:36.576 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-10 06:36:23.567020236 +0000 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-10 06:30:25.101861214 +0000 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-10 06:30:25.101861214 +0000 2024-02-10T06:36:36.634 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-02-10 06:30:25.101861214 +0000 2024-02-10T06:36:36.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T06:36:36.703 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-10T06:36:36.703 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-10T06:36:36.703 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000229374 s, 2.2 MB/s 2024-02-10T06:36:36.704 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T06:36:36.761 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-10 06:36:23.567020236 +0000 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-10 06:30:25.442854684 +0000 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-10 06:30:25.442854684 +0000 2024-02-10T06:36:36.818 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-02-10 06:30:25.442854684 +0000 2024-02-10T06:36:36.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T06:36:36.884 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-10T06:36:36.884 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-10T06:36:36.885 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000207141 s, 2.5 MB/s 2024-02-10T06:36:36.886 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T06:36:36.943 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-10 06:36:23.568020217 +0000 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-10 06:30:25.781848192 +0000 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-10 06:30:25.781848192 +0000 2024-02-10T06:36:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-02-10 06:30:25.781848192 +0000 2024-02-10T06:36:36.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: Reconfiguring mgr.smithi073.vjivfn (unknown last config time)... 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: Reconfiguring daemon mgr.smithi073.vjivfn on smithi073 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:36:37.062 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:36 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:37.064 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-10T06:36:37.064 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-10T06:36:37.064 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000209692 s, 2.4 MB/s 2024-02-10T06:36:37.065 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T06:36:37.121 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-02-10T06:36:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: Reconfiguring mgr.smithi073.vjivfn (unknown last config time)... 2024-02-10T06:36:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: Reconfiguring daemon mgr.smithi073.vjivfn on smithi073 2024-02-10T06:36:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:37.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:37.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:36:37.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:36 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:37.175 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-10 06:36:23.568020217 +0000 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-10 06:30:26.155841030 +0000 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-10 06:30:26.155841030 +0000 2024-02-10T06:36:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-02-10 06:30:26.155841030 +0000 2024-02-10T06:36:37.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T06:36:37.241 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-10T06:36:37.241 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-10T06:36:37.241 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000251934 s, 2.0 MB/s 2024-02-10T06:36:37.242 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T06:36:37.298 INFO:tasks.cephadm:Deploying osd.0 on smithi073 with /dev/vg_nvme/lv_4... 2024-02-10T06:36:37.298 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T06:36:38.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:37 smithi181 ceph-mon[36721]: Reconfiguring crash.smithi073 (monmap changed)... 2024-02-10T06:36:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:37 smithi181 ceph-mon[36721]: Reconfiguring daemon crash.smithi073 on smithi073 2024-02-10T06:36:38.364 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:37 smithi073 ceph-mon[29063]: Reconfiguring crash.smithi073 (monmap changed)... 2024-02-10T06:36:38.364 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:37 smithi073 ceph-mon[29063]: Reconfiguring daemon crash.smithi073 on smithi073 2024-02-10T06:36:38.412 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:36:39.254 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:38 smithi073 ceph-mon[29063]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:39.254 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:38 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:39.254 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:38 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:39.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:38 smithi181 ceph-mon[36721]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:39.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:38 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:39.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:38 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:40.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:39 smithi181 ceph-mon[36721]: Reconfiguring alertmanager.smithi073 (dependencies changed)... 2024-02-10T06:36:40.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:39 smithi181 ceph-mon[36721]: Reconfiguring daemon alertmanager.smithi073 on smithi073 2024-02-10T06:36:40.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:39 smithi073 ceph-mon[29063]: Reconfiguring alertmanager.smithi073 (dependencies changed)... 2024-02-10T06:36:40.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:39 smithi073 ceph-mon[29063]: Reconfiguring daemon alertmanager.smithi073 on smithi073 2024-02-10T06:36:41.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:40 smithi073 ceph-mon[29063]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:41.151 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T06:36:41.152 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T06:36:41.152 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-02-10T06:36:41.152 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-02-10T06:36:41.152 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114295 s, 917 MB/s 2024-02-10T06:36:41.152 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-02-10T06:36:41.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:40 smithi181 ceph-mon[36721]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:42.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:42 smithi181 ceph-mon[36721]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:42.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:42 smithi073 ceph-mon[29063]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:42.758 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_4 2024-02-10T06:36:43.347 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:36:43.469 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:43.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:44.501 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:44 smithi073 ceph-mon[29063]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:44.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:44 smithi181 ceph-mon[36721]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:45.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 -- 172.21.15.73:0/3120973880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 msgr2=0x7f72800fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:45.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 --2- 172.21.15.73:0/3120973880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f72800fd8f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7270004940 tx=0x7f7270009e30 comp rx=0 tx=0).stop 2024-02-10T06:36:45.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 -- 172.21.15.73:0/3120973880 shutdown_connections 2024-02-10T06:36:45.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 --2- 172.21.15.73:0/3120973880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72800fc160 0x7f72800fc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 --2- 172.21.15.73:0/3120973880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f72800fd8f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 -- 172.21.15.73:0/3120973880 >> 172.21.15.73:0/3120973880 conn(0x7f72800f6ca0 msgr2=0x7f72800f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 -- 172.21.15.73:0/3120973880 shutdown_connections 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 -- 172.21.15.73:0/3120973880 wait complete. 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 Processor -- start 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.431+0000 7f7284b13700 1 -- start start 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f7284b13700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72800fc160 0x7f728010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f7284b13700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f728010df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f7284b13700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728010e520 con 0x7f72800fd4e0 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f7284b13700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728010edc0 con 0x7f72800fc160 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f728010df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f728010df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53922/0 (socket says 172.21.15.73:53922) 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 -- 172.21.15.73:0/3968602243 learned_addr learned my addr 172.21.15.73:0/3968602243 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727e59c700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72800fc160 0x7f728010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 -- 172.21.15.73:0/3968602243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72800fc160 msgr2=0x7f728010d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:36:45.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72800fc160 0x7f728010d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 -- 172.21.15.73:0/3968602243 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270005040 con 0x7f72800fd4e0 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f727dd9b700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f728010df20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f727400c830 tx=0x7f7274004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72740106f0 con 0x7f72800fd4e0 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7274010d20 con 0x7f72800fd4e0 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72740113a0 con 0x7f72800fd4e0 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728010f0a0 con 0x7f72800fd4e0 2024-02-10T06:36:45.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.432+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728010f5e0 con 0x7f72800fd4e0 2024-02-10T06:36:45.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.433+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728004bed0 con 0x7f72800fd4e0 2024-02-10T06:36:45.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.434+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f7274012400 con 0x7f72800fd4e0 2024-02-10T06:36:45.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.434+0000 7f726effd700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7268069210 0x7f726806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:36:45.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.434+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7274086040 con 0x7f72800fd4e0 2024-02-10T06:36:45.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.434+0000 7f727e59c700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7268069210 0x7f726806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:36:45.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.435+0000 7f727e59c700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7268069210 0x7f726806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7270004f90 tx=0x7f7270009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:36:45.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.439+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7274056ee0 con 0x7f72800fd4e0 2024-02-10T06:36:45.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:36:45.634+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f728010fa90 con 0x7f7268069210 2024-02-10T06:36:46.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:46 smithi073 ceph-mon[29063]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:46.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:46 smithi073 ceph-mon[29063]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:36:46.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:46 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:36:46.747 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:46 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:36:46.747 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:46 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:46.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:46 smithi181 ceph-mon[36721]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:46.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:46 smithi181 ceph-mon[36721]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:36:46.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:46 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:36:46.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:46 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:36:46.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:46 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:49.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:48 smithi181 ceph-mon[36721]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:49.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:48 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:49.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:48 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:49.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:48 smithi181 ceph-mon[36721]: Reconfiguring grafana.smithi073 (dependencies changed)... 2024-02-10T06:36:49.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:48 smithi181 ceph-mon[36721]: Reconfiguring daemon grafana.smithi073 on smithi073 2024-02-10T06:36:49.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:48 smithi073 ceph-mon[29063]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:49.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:48 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:49.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:48 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:49.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:48 smithi073 ceph-mon[29063]: Reconfiguring grafana.smithi073 (dependencies changed)... 2024-02-10T06:36:49.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:48 smithi073 ceph-mon[29063]: Reconfiguring daemon grafana.smithi073 on smithi073 2024-02-10T06:36:50.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:49 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1552607935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e333d6f-68ed-41e7-a417-bed0ad6e2ce0"}]: dispatch 2024-02-10T06:36:50.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:49 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1552607935' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e333d6f-68ed-41e7-a417-bed0ad6e2ce0"}]': finished 2024-02-10T06:36:50.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:49 smithi181 ceph-mon[36721]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T06:36:50.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:49 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:36:50.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:49 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1320546426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:36:50.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:49 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1552607935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e333d6f-68ed-41e7-a417-bed0ad6e2ce0"}]: dispatch 2024-02-10T06:36:50.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:49 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1552607935' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e333d6f-68ed-41e7-a417-bed0ad6e2ce0"}]': finished 2024-02-10T06:36:50.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:49 smithi073 ceph-mon[29063]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T06:36:50.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:49 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:36:50.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:49 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1320546426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:36:51.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:50 smithi181 ceph-mon[36721]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:51.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:50 smithi073 ceph-mon[29063]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:52.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:52 smithi181 ceph-mon[36721]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:52 smithi073 ceph-mon[29063]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:54.575 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:54 smithi073 ceph-mon[29063]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:54.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:54 smithi181 ceph-mon[36721]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:56.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:56 smithi181 ceph-mon[36721]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:56.851 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:56 smithi073 ceph-mon[29063]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:57.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:57.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:58.683 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:58 smithi073 ceph-mon[29063]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:58.683 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:58.683 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:58.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:58 smithi181 ceph-mon[36721]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:36:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:36:59.613 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:59 smithi073 ceph-mon[29063]: Reconfiguring prometheus.smithi073 (dependencies changed)... 2024-02-10T06:36:59.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:36:59 smithi073 ceph-mon[29063]: Reconfiguring daemon prometheus.smithi073 on smithi073 2024-02-10T06:36:59.760 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:59 smithi181 ceph-mon[36721]: Reconfiguring prometheus.smithi073 (dependencies changed)... 2024-02-10T06:36:59.760 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:36:59 smithi181 ceph-mon[36721]: Reconfiguring daemon prometheus.smithi073 on smithi073 2024-02-10T06:37:00.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:00 smithi181 ceph-mon[36721]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:00.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:00 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:37:00.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:00 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:00.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:00 smithi181 ceph-mon[36721]: Deploying daemon osd.0 on smithi073 2024-02-10T06:37:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:00 smithi073 ceph-mon[29063]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:00 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:37:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:00 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:00 smithi073 ceph-mon[29063]: Deploying daemon osd.0 on smithi073 2024-02-10T06:37:02.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:02 smithi181 ceph-mon[36721]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:02.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:02 smithi073 ceph-mon[29063]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:04.591 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:04 smithi073 ceph-mon[29063]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:04.591 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:04.592 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:04.592 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi181", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:37:04.592 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:04.803 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:04 smithi181 ceph-mon[36721]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:04.803 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:04.803 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:04.803 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi181", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:37:04.804 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:05.596 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: Reconfiguring crash.smithi181 (monmap changed)... 2024-02-10T06:37:05.597 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: Reconfiguring daemon crash.smithi181 on smithi181 2024-02-10T06:37:05.597 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:05.597 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:05.597 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi181.lmidhc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:37:05.597 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:37:05.597 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:05.811 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: Reconfiguring crash.smithi181 (monmap changed)... 2024-02-10T06:37:05.811 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: Reconfiguring daemon crash.smithi181 on smithi181 2024-02-10T06:37:05.811 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:05.811 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:05.811 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi181.lmidhc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:37:05.811 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:37:05.812 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:06.678 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:06 smithi073 ceph-mon[29063]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:06.678 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:06 smithi073 ceph-mon[29063]: Reconfiguring mgr.smithi181.lmidhc (monmap changed)... 2024-02-10T06:37:06.678 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:06 smithi073 ceph-mon[29063]: Reconfiguring daemon mgr.smithi181.lmidhc on smithi181 2024-02-10T06:37:06.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:06 smithi181 ceph-mon[36721]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:06.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:06 smithi181 ceph-mon[36721]: Reconfiguring mgr.smithi181.lmidhc (monmap changed)... 2024-02-10T06:37:06.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:06 smithi181 ceph-mon[36721]: Reconfiguring daemon mgr.smithi181.lmidhc on smithi181 2024-02-10T06:37:08.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:08.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:08.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: Reconfiguring mon.smithi181 (monmap changed)... 2024-02-10T06:37:08.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:37:08.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:37:08.022 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:08.022 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:07 smithi073 ceph-mon[29063]: Reconfiguring daemon mon.smithi181 on smithi181 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: Reconfiguring mon.smithi181 (monmap changed)... 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:07 smithi181 ceph-mon[36721]: Reconfiguring daemon mon.smithi181 on smithi181 2024-02-10T06:37:09.055 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:09.055 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.055 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:37:09.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:37:09.057 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:37:09.057 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.057 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:37:09.150 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:09.151 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:10.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:10 smithi073 ceph-mon[29063]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:10.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:10 smithi181 ceph-mon[36721]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:10.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:10.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:12.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:12 smithi181 ceph-mon[36721]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:12.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:12 smithi073 ceph-mon[29063]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:13.440 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:13.511 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 0 on host 'smithi073' 2024-02-10T06:37:13.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.510+0000 7f726effd700 1 -- 172.21.15.73:0/3968602243 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f728010fa90 con 0x7f7268069210 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7268069210 msgr2=0x7f726806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7268069210 0x7f726806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7270004f90 tx=0x7f7270009e30 comp rx=0 tx=0).stop 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 msgr2=0x7f728010df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f728010df20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f727400c830 tx=0x7f7274004470 comp rx=0 tx=0).stop 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 shutdown_connections 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7268069210 0x7f726806b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72800fc160 0x7f728010d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 --2- 172.21.15.73:0/3968602243 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72800fd4e0 0x7f728010df20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 >> 172.21.15.73:0/3968602243 conn(0x7f72800f6ca0 msgr2=0x7f72800fb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 shutdown_connections 2024-02-10T06:37:13.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:13.513+0000 7f7284b13700 1 -- 172.21.15.73:0/3968602243 wait complete. 2024-02-10T06:37:13.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:13 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:14.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:14 smithi073 ceph-mon[29063]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:37:14.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:14 smithi073 ceph-mon[29063]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:14.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:14 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:14.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:14 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:14.369 DEBUG:teuthology.orchestra.run.smithi073:osd.0> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T06:37:14.372 INFO:tasks.cephadm:Deploying osd.1 on smithi073 with /dev/vg_nvme/lv_3... 2024-02-10T06:37:14.372 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T06:37:14.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:14 smithi181 ceph-mon[36721]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:37:14.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:14 smithi181 ceph-mon[36721]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:14.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:14 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:14.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:14 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:14.805 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:37:15.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:15 smithi181 ceph-mon[36721]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:37:15.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:15 smithi181 ceph-mon[36721]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T06:37:15.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:15 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:15.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:15 smithi181 ceph-mon[36721]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-02-10T06:37:15.604 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:15 smithi073 ceph-mon[29063]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:37:15.604 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:15 smithi073 ceph-mon[29063]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T06:37:15.604 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:15 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:15.604 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:15 smithi073 ceph-mon[29063]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-02-10T06:37:15.605 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:37:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:37:15.303+0000 7feb60ccc700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T06:37:15.605 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:37:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:37:15.307+0000 7feb5bac7700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:37:16.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:16 smithi073 ceph-mon[29063]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-02-10T06:37:16.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:16 smithi073 ceph-mon[29063]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T06:37:16.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:16.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:16.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:16 smithi073 ceph-mon[29063]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:16.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:16.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:16 smithi181 ceph-mon[36721]: from='osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-02-10T06:37:16.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:16 smithi181 ceph-mon[36721]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T06:37:16.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:16.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:16.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:16 smithi181 ceph-mon[36721]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:16.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:17.515 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: purged_snaps scrub starts 2024-02-10T06:37:17.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: purged_snaps scrub ok 2024-02-10T06:37:17.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] boot 2024-02-10T06:37:17.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T06:37:17.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:17.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:17.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:17.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: purged_snaps scrub starts 2024-02-10T06:37:17.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: purged_snaps scrub ok 2024-02-10T06:37:17.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: osd.0 [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] boot 2024-02-10T06:37:17.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T06:37:17.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:37:17.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:17.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:17.866 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T06:37:17.866 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T06:37:17.866 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-02-10T06:37:17.867 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-02-10T06:37:17.867 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144497 s, 726 MB/s 2024-02-10T06:37:17.867 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-02-10T06:37:18.510 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_3 2024-02-10T06:37:18.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:18 smithi181 ceph-mon[36721]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:18.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:18 smithi073 ceph-mon[29063]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:19.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:19 smithi073 ceph-mon[29063]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T06:37:19.504 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:37:19.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:19 smithi181 ceph-mon[36721]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T06:37:20.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:20 smithi073 ceph-mon[29063]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:20.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:20 smithi181 ceph-mon[36721]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:22.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: Detected new or changed devices on smithi073 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 -- 172.21.15.73:0/2191816724 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4100120 msgr2=0x7ff5f4102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/2191816724 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4100120 0x7ff5f4102520 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4004940 tx=0x7ff5e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 -- 172.21.15.73:0/2191816724 shutdown_connections 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/2191816724 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4100120 0x7ff5f4102520 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/2191816724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f4066380 0x7ff5f40ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 -- 172.21.15.73:0/2191816724 >> 172.21.15.73:0/2191816724 conn(0x7ff5f40f74e0 msgr2=0x7ff5f40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 -- 172.21.15.73:0/2191816724 shutdown_connections 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 -- 172.21.15.73:0/2191816724 wait complete. 2024-02-10T06:37:22.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.496+0000 7ff5fb41e700 1 Processor -- start 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5fb41e700 1 -- start start 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5fb41e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 0x7ff5f410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5fb41e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f4100120 0x7ff5f410de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5fb41e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f410e510 con 0x7ff5f4066380 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5fb41e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f410ed50 con 0x7ff5f4100120 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 0x7ff5f410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 0x7ff5f410d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51760/0 (socket says 172.21.15.73:51760) 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 -- 172.21.15.73:0/212238750 learned_addr learned my addr 172.21.15.73:0/212238750 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f89b9700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f4100120 0x7ff5f410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 -- 172.21.15.73:0/212238750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f4100120 msgr2=0x7ff5f410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f4100120 0x7ff5f410de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 -- 172.21.15.73:0/212238750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e4005040 con 0x7ff5f4066380 2024-02-10T06:37:22.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5f91ba700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 0x7ff5f410d940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000b790 tx=0x7ff5f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5f000f770 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f410f030 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.497+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5f000fda0 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.498+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5f000d310 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.498+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f410f570 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.499+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f4107ad0 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.499+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7ff5f0010420 con 0x7ff5f4066380 2024-02-10T06:37:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.500+0000 7ff5e9ffb700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff5e0069210 0x7ff5e006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:37:22.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.500+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7ff5f00862f0 con 0x7ff5f4066380 2024-02-10T06:37:22.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.500+0000 7ff5f89b9700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff5e0069210 0x7ff5e006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:37:22.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.500+0000 7ff5f89b9700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff5e0069210 0x7ff5e006b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400b3b0 tx=0x7ff5e4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:37:22.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.504+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff5f0056f30 con 0x7ff5f4066380 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: Detected new or changed devices on smithi073 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:22.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:22.718+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5f410f890 con 0x7ff5e0069210 2024-02-10T06:37:23.588 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:23 smithi073 ceph-mon[29063]: Adjusting osd_memory_target on smithi073 to 12864M 2024-02-10T06:37:23.588 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:23 smithi073 ceph-mon[29063]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:23.588 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:23 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:37:23.588 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:23 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:37:23.588 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:23 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:23.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:23 smithi181 ceph-mon[36721]: Adjusting osd_memory_target on smithi073 to 12864M 2024-02-10T06:37:23.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:23 smithi181 ceph-mon[36721]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:23.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:23 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:37:23.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:23 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:37:23.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:23 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:25.018 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:24 smithi073 ceph-mon[29063]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:25.018 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:24 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:25.018 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:24 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:25.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:24 smithi181 ceph-mon[36721]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:25.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:24 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:25.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:24 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:27.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:26 smithi181 ceph-mon[36721]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:27.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:26 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3645449134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dedbb82b-7b4d-47eb-8621-2a35fa20a05d"}]: dispatch 2024-02-10T06:37:27.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:26 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3645449134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dedbb82b-7b4d-47eb-8621-2a35fa20a05d"}]': finished 2024-02-10T06:37:27.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:26 smithi181 ceph-mon[36721]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T06:37:27.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:26 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:26 smithi073 ceph-mon[29063]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:26 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3645449134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dedbb82b-7b4d-47eb-8621-2a35fa20a05d"}]: dispatch 2024-02-10T06:37:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:26 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3645449134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dedbb82b-7b4d-47eb-8621-2a35fa20a05d"}]': finished 2024-02-10T06:37:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:26 smithi073 ceph-mon[29063]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T06:37:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:26 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:28.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:28.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:27 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1722987104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:37:28.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:28.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:28.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:28.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:28.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:28.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:27 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1722987104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:37:28.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:28.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:28.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:28.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:29.242 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:28 smithi181 ceph-mon[36721]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:29.242 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:28 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:29.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:28 smithi073 ceph-mon[29063]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:29.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:28 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:31.094 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:30 smithi073 ceph-mon[29063]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:31.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:30 smithi181 ceph-mon[36721]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:32.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:32 smithi073 ceph-mon[29063]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:32.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:32 smithi181 ceph-mon[36721]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:34.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:34 smithi073 ceph-mon[29063]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:34.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:34 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:37:34.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:34 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:34.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:34 smithi181 ceph-mon[36721]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:34.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:34 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:37:34.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:34 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:35.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:35 smithi181 ceph-mon[36721]: Deploying daemon osd.1 on smithi073 2024-02-10T06:37:35.848 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:35 smithi073 ceph-mon[29063]: Deploying daemon osd.1 on smithi073 2024-02-10T06:37:36.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:36 smithi181 ceph-mon[36721]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:36.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:36 smithi073 ceph-mon[29063]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:38 smithi073 ceph-mon[29063]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:38.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:38 smithi181 ceph-mon[36721]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:40.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:40 smithi181 ceph-mon[36721]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:40.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:40 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:40.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:40 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:40.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:40 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:40 smithi073 ceph-mon[29063]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:40 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:40 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:40 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:42.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:42.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:42.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:42 smithi073 ceph-mon[29063]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:42.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:42.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:42.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:42 smithi181 ceph-mon[36721]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:43.123 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 1 on host 'smithi073' 2024-02-10T06:37:43.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.122+0000 7ff5e9ffb700 1 -- 172.21.15.73:0/212238750 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5f410f890 con 0x7ff5e0069210 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff5e0069210 msgr2=0x7ff5e006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff5e0069210 0x7ff5e006b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400b3b0 tx=0x7ff5e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 msgr2=0x7ff5f410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 0x7ff5f410d940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000b790 tx=0x7ff5f0009250 comp rx=0 tx=0).stop 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 shutdown_connections 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff5e0069210 0x7ff5e006b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5f4066380 0x7ff5f410d940 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:43.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 --2- 172.21.15.73:0/212238750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f4100120 0x7ff5f410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:43.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 >> 172.21.15.73:0/212238750 conn(0x7ff5f40f74e0 msgr2=0x7ff5f40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:37:43.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 shutdown_connections 2024-02-10T06:37:43.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:43.124+0000 7ff5fb41e700 1 -- 172.21.15.73:0/212238750 wait complete. 2024-02-10T06:37:43.365 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:43.365 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:43.365 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:43.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:43.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:43.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:44.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:44 smithi181 ceph-mon[36721]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:37:44.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:44 smithi181 ceph-mon[36721]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:44.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:44 smithi073 ceph-mon[29063]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:37:44.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:44 smithi073 ceph-mon[29063]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:44.750 DEBUG:teuthology.orchestra.run.smithi073:osd.1> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T06:37:44.752 INFO:tasks.cephadm:Deploying osd.2 on smithi073 with /dev/vg_nvme/lv_2... 2024-02-10T06:37:44.752 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T06:37:45.484 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:37:45.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:45 smithi181 ceph-mon[36721]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:37:45.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:45 smithi181 ceph-mon[36721]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T06:37:45.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:45 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:45.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:45 smithi181 ceph-mon[36721]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-02-10T06:37:45.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:45 smithi073 ceph-mon[29063]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:37:45.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:45 smithi073 ceph-mon[29063]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T06:37:45.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:45 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:45.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:45 smithi073 ceph-mon[29063]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-02-10T06:37:45.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:37:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:37:45.298+0000 7f725654d700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T06:37:45.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:37:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:37:45.302+0000 7f7251b49700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:37:46.409 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:46 smithi073 ceph-mon[29063]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-02-10T06:37:46.410 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:46 smithi073 ceph-mon[29063]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T06:37:46.410 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:46 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:46.410 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:46 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:46.410 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:46 smithi073 ceph-mon[29063]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:46.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:46 smithi181 ceph-mon[36721]: from='osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-02-10T06:37:46.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:46 smithi181 ceph-mon[36721]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T06:37:46.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:46 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:46.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:46 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:46.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:46 smithi181 ceph-mon[36721]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:37:47.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:47 smithi073 ceph-mon[29063]: purged_snaps scrub starts 2024-02-10T06:37:47.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:47 smithi073 ceph-mon[29063]: purged_snaps scrub ok 2024-02-10T06:37:47.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:47 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:47.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:47 smithi073 ceph-mon[29063]: osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] boot 2024-02-10T06:37:47.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:47 smithi073 ceph-mon[29063]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T06:37:47.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:47 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:47.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:47 smithi181 ceph-mon[36721]: purged_snaps scrub starts 2024-02-10T06:37:47.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:47 smithi181 ceph-mon[36721]: purged_snaps scrub ok 2024-02-10T06:37:47.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:47 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:47.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:47 smithi181 ceph-mon[36721]: osd.1 [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] boot 2024-02-10T06:37:47.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:47 smithi181 ceph-mon[36721]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T06:37:47.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:47 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:37:48.140 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T06:37:48.141 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T06:37:48.141 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-02-10T06:37:48.141 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-02-10T06:37:48.141 INFO:teuthology.orchestra.run.smithi073.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013853 s, 757 MB/s 2024-02-10T06:37:48.141 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-02-10T06:37:48.512 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:48 smithi073 ceph-mon[29063]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T06:37:48.512 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:48 smithi073 ceph-mon[29063]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:48.512 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:48 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:48.512 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:48 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:48.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:48 smithi181 ceph-mon[36721]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T06:37:48.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:48 smithi181 ceph-mon[36721]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:48.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:48 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:48.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:48 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:49.527 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_2 2024-02-10T06:37:50.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:50 smithi073 ceph-mon[29063]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:50.648 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:37:50.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:50 smithi181 ceph-mon[36721]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:52.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:52.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:52.614 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:52.615 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:52.615 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:52.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 -- 172.21.15.73:0/1679086964 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 msgr2=0x7f09e80fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:52.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/1679086964 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e80fd340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f09d8004940 tx=0x7f09d8009e30 comp rx=0 tx=0).stop 2024-02-10T06:37:52.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 -- 172.21.15.73:0/1679086964 shutdown_connections 2024-02-10T06:37:52.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/1679086964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 0x7f09e80fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:52.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/1679086964 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e80fd340 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 -- 172.21.15.73:0/1679086964 >> 172.21.15.73:0/1679086964 conn(0x7f09e80f86b0 msgr2=0x7f09e80faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 -- 172.21.15.73:0/1679086964 shutdown_connections 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.951+0000 7f09ee3a0700 1 -- 172.21.15.73:0/1679086964 wait complete. 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.952+0000 7f09ee3a0700 1 Processor -- start 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.952+0000 7f09ee3a0700 1 -- start start 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.952+0000 7f09ee3a0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.952+0000 7f09ee3a0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 0x7f09e810dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.952+0000 7f09ee3a0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e810e4a0 con 0x7f09e80fcf30 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.952+0000 7f09ee3a0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e810ed40 con 0x7f09e80fe1e0 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e810d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43402/0 (socket says 172.21.15.73:43402) 2024-02-10T06:37:52.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 -- 172.21.15.73:0/295199348 learned_addr learned my addr 172.21.15.73:0/295199348 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e77fe700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 0x7f09e810dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 -- 172.21.15.73:0/295199348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 msgr2=0x7f09e810dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 0x7f09e810dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 -- 172.21.15.73:0/295199348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d8005040 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e77fe700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 0x7f09e810dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e7fff700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e810d960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f09d80093b0 tx=0x7f09d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09d800e040 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09d800bcc0 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e810efc0 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09d801c630 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.953+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e810f500 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.954+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e804bed0 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.955+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f09d80139f0 con 0x7f09e80fcf30 2024-02-10T06:37:52.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.955+0000 7f09e4ff9700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f09d0069210 0x7f09d006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:37:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.955+0000 7f09e77fe700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f09d0069210 0x7f09d006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:37:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.955+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f09d80878e0 con 0x7f09e80fcf30 2024-02-10T06:37:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.955+0000 7f09e77fe700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f09d0069210 0x7f09d006b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009fd0 tx=0x7f09dc009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:37:52.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:52.960+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f09d8058370 con 0x7f09e80fcf30 2024-02-10T06:37:53.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:37:53.155+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f09e8102b30 con 0x7f09d0069210 2024-02-10T06:37:53.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:53 smithi181 ceph-mon[36721]: Detected new or changed devices on smithi073 2024-02-10T06:37:53.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:53 smithi181 ceph-mon[36721]: Adjusting osd_memory_target on smithi073 to 6432M 2024-02-10T06:37:53.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:53 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:37:53.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:53 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:37:53.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:53 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:53.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:53 smithi073 ceph-mon[29063]: Detected new or changed devices on smithi073 2024-02-10T06:37:53.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:53 smithi073 ceph-mon[29063]: Adjusting osd_memory_target on smithi073 to 6432M 2024-02-10T06:37:53.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:53 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:37:53.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:53 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:37:53.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:53 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:54.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:54 smithi181 ceph-mon[36721]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:54.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:54 smithi181 ceph-mon[36721]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:54.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:54 smithi073 ceph-mon[29063]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:54.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:54 smithi073 ceph-mon[29063]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:56.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:56 smithi181 ceph-mon[36721]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:56.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:56 smithi073 ceph-mon[29063]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:57.670 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:57 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3467363303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e7d61ae-a65b-43db-ac2d-aec724834f40"}]: dispatch 2024-02-10T06:37:57.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:57 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3467363303' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e7d61ae-a65b-43db-ac2d-aec724834f40"}]': finished 2024-02-10T06:37:57.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:57 smithi073 ceph-mon[29063]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T06:37:57.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:37:57.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:57 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3535682774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:37:57.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:57.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:57 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3467363303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e7d61ae-a65b-43db-ac2d-aec724834f40"}]: dispatch 2024-02-10T06:37:57.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:57 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3467363303' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e7d61ae-a65b-43db-ac2d-aec724834f40"}]': finished 2024-02-10T06:37:57.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:57 smithi181 ceph-mon[36721]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T06:37:57.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:37:57.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:57 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3535682774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:37:57.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:37:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:58 smithi181 ceph-mon[36721]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:37:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:58 smithi073 ceph-mon[29063]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:37:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:37:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:37:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:37:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:00.809 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:00 smithi073 ceph-mon[29063]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:00.912 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:00 smithi181 ceph-mon[36721]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:02.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:02 smithi073 ceph-mon[29063]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:02.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:02 smithi181 ceph-mon[36721]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:04.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:04 smithi073 ceph-mon[29063]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:04.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:38:04.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:04.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:04 smithi073 ceph-mon[29063]: Deploying daemon osd.2 on smithi073 2024-02-10T06:38:04.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:04 smithi181 ceph-mon[36721]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:38:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:04 smithi181 ceph-mon[36721]: Deploying daemon osd.2 on smithi073 2024-02-10T06:38:06.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:06 smithi073 ceph-mon[29063]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:06.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:06 smithi181 ceph-mon[36721]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:08.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:08 smithi073 ceph-mon[29063]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:08.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:08 smithi181 ceph-mon[36721]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:10.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:10.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:10.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:10 smithi073 ceph-mon[29063]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:10.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:10.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:10.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:10.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:10 smithi181 ceph-mon[36721]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:10.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:12.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:12 smithi181 ceph-mon[36721]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:12.623 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:12 smithi073 ceph-mon[29063]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:13.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.162+0000 7f09e4ff9700 1 -- 172.21.15.73:0/295199348 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f09e8102b30 con 0x7f09d0069210 2024-02-10T06:38:13.164 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 2 on host 'smithi073' 2024-02-10T06:38:13.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f09d0069210 msgr2=0x7f09d006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:13.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f09d0069210 0x7f09d006b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009fd0 tx=0x7f09dc009ee0 comp rx=0 tx=0).stop 2024-02-10T06:38:13.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 msgr2=0x7f09e810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:13.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e810d960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f09d80093b0 tx=0x7f09d8009e30 comp rx=0 tx=0).stop 2024-02-10T06:38:13.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 shutdown_connections 2024-02-10T06:38:13.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f09d0069210 0x7f09d006b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:13.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09e80fcf30 0x7f09e810d960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:13.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 --2- 172.21.15.73:0/295199348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09e80fe1e0 0x7f09e810dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:13.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.164+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 >> 172.21.15.73:0/295199348 conn(0x7f09e80f86b0 msgr2=0x7f09e8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:38:13.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.165+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 shutdown_connections 2024-02-10T06:38:13.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:38:13.165+0000 7f09ee3a0700 1 -- 172.21.15.73:0/295199348 wait complete. 2024-02-10T06:38:13.408 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:13.408 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:13.408 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:13 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:13.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:13 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:13.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:13 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:13.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:13 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:14.104 DEBUG:teuthology.orchestra.run.smithi073:osd.2> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T06:38:14.107 INFO:tasks.cephadm:Deploying osd.3 on smithi181 with /dev/vg_nvme/lv_4... 2024-02-10T06:38:14.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T06:38:14.365 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:38:14.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:14 smithi073 ceph-mon[29063]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:38:14.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:14 smithi073 ceph-mon[29063]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:14.486 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:14 smithi181 ceph-mon[36721]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:38:14.486 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:14 smithi181 ceph-mon[36721]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:15.525 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:15 smithi181 ceph-mon[36721]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:38:15.525 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:15 smithi181 ceph-mon[36721]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T06:38:15.525 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:15 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:15.525 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:15 smithi181 ceph-mon[36721]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-02-10T06:38:15.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:15 smithi073 ceph-mon[29063]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:38:15.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:15 smithi073 ceph-mon[29063]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T06:38:15.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:15 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:15.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:15 smithi073 ceph-mon[29063]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-02-10T06:38:15.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:38:15.308+0000 7fabe3e26700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T06:38:15.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:38:15.313+0000 7fabdbc1b700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113953 s, 920 MB/s 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 2024-02-10T06:38:16.193 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-10T06:38:16.495 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:16 smithi181 ceph-mon[36721]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-02-10T06:38:16.495 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:16 smithi181 ceph-mon[36721]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T06:38:16.495 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:16.495 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:16.495 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:16 smithi181 ceph-mon[36721]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:16.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:16 smithi073 ceph-mon[29063]: from='osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-02-10T06:38:16.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:16 smithi073 ceph-mon[29063]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T06:38:16.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:16.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:16.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:16 smithi073 ceph-mon[29063]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:38:16.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-02-10T06:38:17.062 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:38:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: purged_snaps scrub starts 2024-02-10T06:38:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: purged_snaps scrub ok 2024-02-10T06:38:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] boot 2024-02-10T06:38:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T06:38:17.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:17.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:17.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: purged_snaps scrub starts 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: purged_snaps scrub ok 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: osd.2 [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] boot 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:17.442 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:18.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:18 smithi073 ceph-mon[29063]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T06:38:18.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:18 smithi073 ceph-mon[29063]: pgmap v78: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:18.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:18 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T06:38:18.423 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- 172.21.15.181:0/2330749118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 msgr2=0x7fa1cc0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:18.423 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 --2- 172.21.15.181:0/2330749118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc0fd420 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc004940 tx=0x7fa1bc009e30 comp rx=0 tx=0).stop 2024-02-10T06:38:18.423 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- 172.21.15.181:0/2330749118 shutdown_connections 2024-02-10T06:38:18.423 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 --2- 172.21.15.181:0/2330749118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc0fd420 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:18.423 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 --2- 172.21.15.181:0/2330749118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1cc0fbe40 0x7fa1cc0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- 172.21.15.181:0/2330749118 >> 172.21.15.181:0/2330749118 conn(0x7fa1cc0f74e0 msgr2=0x7fa1cc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- 172.21.15.181:0/2330749118 shutdown_connections 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- 172.21.15.181:0/2330749118 wait complete. 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 Processor -- start 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- start start 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1cc0fbe40 0x7fa1cc10d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc10dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc10e4a0 con 0x7fa1cc0fbe40 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d3702700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc10ed40 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d0c9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc10dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d0c9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc10dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:53618/0 (socket says 172.21.15.181:53618) 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d0c9d700 1 -- 172.21.15.181:0/1405469647 learned_addr learned my addr 172.21.15.181:0/1405469647 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:38:18.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.421+0000 7fa1d149e700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1cc0fbe40 0x7fa1cc10d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1d0c9d700 1 -- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1cc0fbe40 msgr2=0x7fa1cc10d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1d0c9d700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1cc0fbe40 0x7fa1cc10d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1d0c9d700 1 -- 172.21.15.181:0/1405469647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc005040 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1d0c9d700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc10dea0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0093b0 tx=0x7fa1bc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1bc00e040 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1bc00bcc0 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1cc10efc0 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1bc01c630 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.425 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.422+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1cc10f480 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.423+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1cc04bed0 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.424+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fa1bc0139f0 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.424+0000 7fa1c1ffb700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa1b806d620 0x7fa1b806fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:38:18.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.424+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7fa1bc087c10 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.424+0000 7fa1d149e700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa1b806d620 0x7fa1b806fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:38:18.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.425+0000 7fa1d149e700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa1b806d620 0x7fa1b806fad0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8004990 tx=0x7fa1c8009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:38:18.429 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.429+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa1bc058420 con 0x7fa1cc0fcfb0 2024-02-10T06:38:18.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:18 smithi181 ceph-mon[36721]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T06:38:18.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:18 smithi181 ceph-mon[36721]: pgmap v78: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:18.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:18 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T06:38:18.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:18.621+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1cc10f910 con 0x7fa1b806d620 2024-02-10T06:38:19.571 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T06:38:19.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T06:38:19.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:38:19.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:38:19.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:38:19.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:38:19.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:19 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:19.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T06:38:19.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T06:38:19.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:38:19.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:38:19.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:38:19.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:38:19.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:19 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66197]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:38:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66197]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66197]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:20.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66200]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:38:20.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:20.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66200]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:20.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:38:19 smithi073 sudo[66200]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:20.546 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66301]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:38:20.546 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66301]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:20.546 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66301]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:20.546 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66301]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:20.546 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:38:20.546 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T06:38:20.546 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:20.546 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: Detected new or changed devices on smithi073 2024-02-10T06:38:20.546 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:20 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: Detected new or changed devices on smithi073 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:20.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:20.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:20.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:20.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:20.592 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:20 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:20.806 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66391]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:38:20.806 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66391]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:20.806 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66391]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:20.806 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66391]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:21.022 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:21.020+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa1bc054d60 con 0x7fa1cc0fcfb0 2024-02-10T06:38:21.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66413]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:38:21.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:21.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66413]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:21.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:20 smithi073 sudo[66413]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:21.326 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66416]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:38:21.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66416]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:21.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66416]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:21.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66416]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:21.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66501]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:38:21.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66501]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:21.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66501]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:21.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66501]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:21.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66526]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:38:21.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66526]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:21.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66526]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 sudo[66526]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: Adjusting osd_memory_target on smithi073 to 4288M 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/271228211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f17b271-99d5-4b38-89fb-175414c59351"}]: dispatch 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f17b271-99d5-4b38-89fb-175414c59351"}]: dispatch 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f17b271-99d5-4b38-89fb-175414c59351"}]': finished 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: mgrmap e19: smithi073.vjivfn(active, since 2m), standbys: smithi181.lmidhc 2024-02-10T06:38:21.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:21 smithi073 ceph-mon[29063]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: Adjusting osd_memory_target on smithi073 to 4288M 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/271228211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f17b271-99d5-4b38-89fb-175414c59351"}]: dispatch 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f17b271-99d5-4b38-89fb-175414c59351"}]: dispatch 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f17b271-99d5-4b38-89fb-175414c59351"}]': finished 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:21.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: mgrmap e19: smithi073.vjivfn(active, since 2m), standbys: smithi181.lmidhc 2024-02-10T06:38:21.989 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 ceph-mon[36721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:38:21.989 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 sudo[40287]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:38:21.989 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 sudo[40287]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:21.989 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 sudo[40287]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:21.989 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:21 smithi181 sudo[40287]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:22.590 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 sudo[40313]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:38:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 sudo[40313]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:38:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 sudo[40313]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:38:22.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 sudo[40313]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/2603802713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:38:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:38:22.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/2603802713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mon metadata", "id": "smithi181"}]: dispatch 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:38:23.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:24.858 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:24 smithi181 ceph-mon[36721]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:24.858 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:24 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:24.858 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:24 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:24.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:24 smithi073 ceph-mon[29063]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:24.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:24 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:24.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:24 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:25.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:25 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:25.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:25 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:25.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:25 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:25.899 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:25 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:25.899 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:25 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:25.899 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:25 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:26.696 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:26 smithi181 ceph-mon[36721]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:26 smithi073 ceph-mon[29063]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:27.933 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:28.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:28.872 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:28 smithi181 ceph-mon[36721]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:28.872 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:28 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:38:28.872 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:28 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:28.873 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:28 smithi181 ceph-mon[36721]: Deploying daemon osd.3 on smithi181 2024-02-10T06:38:29.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:28 smithi073 ceph-mon[29063]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:29.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:28 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:38:29.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:28 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:29.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:28 smithi073 ceph-mon[29063]: Deploying daemon osd.3 on smithi181 2024-02-10T06:38:30.960 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:30 smithi181 ceph-mon[36721]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:30 smithi073 ceph-mon[29063]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:32.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:32 smithi073 ceph-mon[29063]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:33.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:32 smithi181 ceph-mon[36721]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:34.201 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:33 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:34.201 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:33 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:34.201 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:33 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:34.313 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:33 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:34.314 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:33 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:34.314 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:33 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:35.186 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:34 smithi181 ceph-mon[36721]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:35.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:34 smithi073 ceph-mon[29063]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:35.930 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.929+0000 7fa1c1ffb700 1 -- 172.21.15.181:0/1405469647 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa1cc10f910 con 0x7fa1b806d620 2024-02-10T06:38:35.930 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 3 on host 'smithi181' 2024-02-10T06:38:35.932 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa1b806d620 msgr2=0x7fa1b806fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:35.932 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa1b806d620 0x7fa1b806fad0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8004990 tx=0x7fa1c8009ee0 comp rx=0 tx=0).stop 2024-02-10T06:38:35.932 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 msgr2=0x7fa1cc10dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:35.932 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc10dea0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0093b0 tx=0x7fa1bc009e30 comp rx=0 tx=0).stop 2024-02-10T06:38:35.932 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 shutdown_connections 2024-02-10T06:38:35.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa1b806d620 0x7fa1b806fad0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:35.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1cc0fbe40 0x7fa1cc10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:35.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 --2- 172.21.15.181:0/1405469647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1cc0fcfb0 0x7fa1cc10dea0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:35.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 >> 172.21.15.181:0/1405469647 conn(0x7fa1cc0f74e0 msgr2=0x7fa1cc0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:38:35.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 shutdown_connections 2024-02-10T06:38:35.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:35.931+0000 7fa1d3702700 1 -- 172.21.15.181:0/1405469647 wait complete. 2024-02-10T06:38:36.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:36 smithi181 ceph-mon[36721]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:36.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:36 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:36.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:36 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:36.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:36 smithi181 ceph-mon[36721]: from='osd.3 [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:38:36.988 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:36 smithi181 ceph-mon[36721]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:38:37.012 DEBUG:teuthology.orchestra.run.smithi181:osd.3> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T06:38:37.014 INFO:tasks.cephadm:Deploying osd.4 on smithi181 with /dev/vg_nvme/lv_3... 2024-02-10T06:38:37.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T06:38:37.313 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:38:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:36 smithi073 ceph-mon[29063]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:36 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:36 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:36 smithi073 ceph-mon[29063]: from='osd.3 [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:38:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:36 smithi073 ceph-mon[29063]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: from='osd.3 [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:38:38.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-mon[36721]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:38:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:38:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T06:38:38.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:38.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: from='osd.3 [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:38:38.622 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:38 smithi073 ceph-mon[29063]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:38:38.707 INFO:journalctl@ceph.osd.3.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-3[42329]: 2024-02-10T06:38:38.602+0000 7fb6d5ca0700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T06:38:38.707 INFO:journalctl@ceph.osd.3.smithi181.stdout:Feb 10 06:38:38 smithi181 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-3[42329]: 2024-02-10T06:38:38.608+0000 7fb6cf298700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:38:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:39 smithi073 ceph-mon[29063]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-10T06:38:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:39 smithi073 ceph-mon[29063]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T06:38:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:39 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:39 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:39.877 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:39 smithi181 ceph-mon[36721]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-10T06:38:39.877 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:39 smithi181 ceph-mon[36721]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T06:38:39.877 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:39 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:39.877 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:39 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:40.157 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T06:38:40.157 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T06:38:40.158 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-10T06:38:40.158 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-10T06:38:40.158 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114097 s, 919 MB/s 2024-02-10T06:38:40.158 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-10T06:38:40.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-02-10T06:38:40.863 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:40 smithi181 ceph-mon[36721]: purged_snaps scrub starts 2024-02-10T06:38:40.863 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:40 smithi181 ceph-mon[36721]: purged_snaps scrub ok 2024-02-10T06:38:40.863 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:40 smithi181 ceph-mon[36721]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:40.864 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:40 smithi181 ceph-mon[36721]: osd.3 [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] boot 2024-02-10T06:38:40.864 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:40 smithi181 ceph-mon[36721]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T06:38:40.864 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:40 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:40 smithi073 ceph-mon[29063]: purged_snaps scrub starts 2024-02-10T06:38:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:40 smithi073 ceph-mon[29063]: purged_snaps scrub ok 2024-02-10T06:38:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:40 smithi073 ceph-mon[29063]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:38:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:40 smithi073 ceph-mon[29063]: osd.3 [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] boot 2024-02-10T06:38:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:40 smithi073 ceph-mon[29063]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T06:38:40.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:40 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:38:41.487 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:38:41.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:41 smithi181 ceph-mon[36721]: osdmap e29: 4 total, 4 up, 4 in 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:41.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:41 smithi073 ceph-mon[29063]: osdmap e29: 4 total, 4 up, 4 in 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: Detected new or changed devices on smithi181 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: Adjusting osd_memory_target on smithi181 to 15936M 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:42.748 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: Detected new or changed devices on smithi181 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: Adjusting osd_memory_target on smithi181 to 15936M 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:42.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 -- 172.21.15.181:0/2079764962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 msgr2=0x7fa6a80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 --2- 172.21.15.181:0/2079764962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a80fd420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa69c004940 tx=0x7fa69c009e30 comp rx=0 tx=0).stop 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 -- 172.21.15.181:0/2079764962 shutdown_connections 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 --2- 172.21.15.181:0/2079764962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a80fd420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 --2- 172.21.15.181:0/2079764962 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa6a80fbe40 0x7fa6a80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 -- 172.21.15.181:0/2079764962 >> 172.21.15.181:0/2079764962 conn(0x7fa6a80f7520 msgr2=0x7fa6a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 -- 172.21.15.181:0/2079764962 shutdown_connections 2024-02-10T06:38:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 -- 172.21.15.181:0/2079764962 wait complete. 2024-02-10T06:38:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.940+0000 7fa6ae323700 1 Processor -- start 2024-02-10T06:38:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 -- start start 2024-02-10T06:38:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa6a80fbe40 0x7fa6a810c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:38:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a810cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:38:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a810d3b0 con 0x7fa6a80fbe40 2024-02-10T06:38:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a810dc50 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a810cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a810cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49438/0 (socket says 172.21.15.181:49438) 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 -- 172.21.15.181:0/1163924201 learned_addr learned my addr 172.21.15.181:0/1163924201 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a7fff700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa6a80fbe40 0x7fa6a810c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 -- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa6a80fbe40 msgr2=0x7fa6a810c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa6a80fbe40 0x7fa6a810c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 -- 172.21.15.181:0/1163924201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c005040 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a77fe700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a810cdb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0093b0 tx=0x7fa69c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69c00e040 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa69c00bcc0 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a810ded0 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69c01c630 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.941+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a810e410 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.942+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a804bed0 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.943+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa69c0139f0 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.943+0000 7fa6a4ff9700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa6900692a0 0x7fa69006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:38:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.943+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fa69c087f40 con 0x7fa6a80fcfb0 2024-02-10T06:38:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.943+0000 7fa6a7fff700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa6900692a0 0x7fa69006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:38:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.944+0000 7fa6a7fff700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa6900692a0 0x7fa69006b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa6980041d0 tx=0x7fa698009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:38:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:42.948+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa69c0b9bd0 con 0x7fa6a80fcfb0 2024-02-10T06:38:43.139 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:38:43.139+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6a810e860 con 0x7fa6900692a0 2024-02-10T06:38:43.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:43 smithi181 ceph-mon[36721]: osdmap e30: 4 total, 4 up, 4 in 2024-02-10T06:38:43.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:38:43.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:38:43.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:43 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:44.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:43 smithi073 ceph-mon[29063]: osdmap e30: 4 total, 4 up, 4 in 2024-02-10T06:38:44.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:38:44.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:38:44.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:43 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:44.826 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:44 smithi181 ceph-mon[36721]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:38:44.826 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:44 smithi181 ceph-mon[36721]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-10T06:38:44.826 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:44 smithi181 ceph-mon[36721]: osdmap e31: 4 total, 4 up, 4 in 2024-02-10T06:38:45.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:44 smithi073 ceph-mon[29063]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:38:45.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:44 smithi073 ceph-mon[29063]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-10T06:38:45.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:44 smithi073 ceph-mon[29063]: osdmap e31: 4 total, 4 up, 4 in 2024-02-10T06:38:46.045 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:45 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/1902934032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28fd87c6-5f23-44af-87a6-d3cf81095f7e"}]: dispatch 2024-02-10T06:38:46.045 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:45 smithi181 ceph-mon[36721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28fd87c6-5f23-44af-87a6-d3cf81095f7e"}]: dispatch 2024-02-10T06:38:46.045 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:45 smithi181 ceph-mon[36721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28fd87c6-5f23-44af-87a6-d3cf81095f7e"}]': finished 2024-02-10T06:38:46.045 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:45 smithi181 ceph-mon[36721]: osdmap e32: 5 total, 4 up, 5 in 2024-02-10T06:38:46.045 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:45 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:38:46.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:45 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1902934032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28fd87c6-5f23-44af-87a6-d3cf81095f7e"}]: dispatch 2024-02-10T06:38:46.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:45 smithi073 ceph-mon[29063]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28fd87c6-5f23-44af-87a6-d3cf81095f7e"}]: dispatch 2024-02-10T06:38:46.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:45 smithi073 ceph-mon[29063]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28fd87c6-5f23-44af-87a6-d3cf81095f7e"}]': finished 2024-02-10T06:38:46.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:45 smithi073 ceph-mon[29063]: osdmap e32: 5 total, 4 up, 5 in 2024-02-10T06:38:46.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:45 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:38:47.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:46 smithi181 ceph-mon[36721]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-10T06:38:47.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:46 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/3523031346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:38:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:46 smithi073 ceph-mon[29063]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-10T06:38:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:46 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3523031346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:38:49.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:48 smithi181 ceph-mon[36721]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T06:38:49.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:48 smithi073 ceph-mon[29063]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T06:38:50.945 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:50 smithi181 ceph-mon[36721]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:51.124 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:50 smithi073 ceph-mon[29063]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:52.673 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:52 smithi181 ceph-mon[36721]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:52.674 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:38:52.674 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:52 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:53.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:52 smithi073 ceph-mon[29063]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:53.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:38:53.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:52 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:53.766 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:53 smithi181 ceph-mon[36721]: Deploying daemon osd.4 on smithi181 2024-02-10T06:38:54.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:53 smithi073 ceph-mon[29063]: Deploying daemon osd.4 on smithi181 2024-02-10T06:38:54.740 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:54 smithi181 ceph-mon[36721]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:55.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:54 smithi073 ceph-mon[29063]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:56.931 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:56 smithi181 ceph-mon[36721]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:57.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:56 smithi073 ceph-mon[29063]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:58 smithi181 ceph-mon[36721]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:58.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:38:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:58 smithi073 ceph-mon[29063]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:38:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:38:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:38:58.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:38:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:00.501 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 4 on host 'smithi181' 2024-02-10T06:39:00.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.500+0000 7fa6a4ff9700 1 -- 172.21.15.181:0/1163924201 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa6a810e860 con 0x7fa6900692a0 2024-02-10T06:39:00.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa6900692a0 msgr2=0x7fa69006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:00.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa6900692a0 0x7fa69006b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa6980041d0 tx=0x7fa698009040 comp rx=0 tx=0).stop 2024-02-10T06:39:00.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 msgr2=0x7fa6a810cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:00.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a810cdb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0093b0 tx=0x7fa69c009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 shutdown_connections 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa6900692a0 0x7fa69006b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa6a80fbe40 0x7fa6a810c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 --2- 172.21.15.181:0/1163924201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6a80fcfb0 0x7fa6a810cdb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 >> 172.21.15.181:0/1163924201 conn(0x7fa6a80f7520 msgr2=0x7fa6a8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 shutdown_connections 2024-02-10T06:39:00.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:00.502+0000 7fa6ae323700 1 -- 172.21.15.181:0/1163924201 wait complete. 2024-02-10T06:39:00.743 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:00 smithi181 ceph-mon[36721]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:39:00.744 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:00 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:00.744 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:00 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:00 smithi073 ceph-mon[29063]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:39:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:00 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:00.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:00 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:01.591 DEBUG:teuthology.orchestra.run.smithi181:osd.4> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T06:39:01.594 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_2... 2024-02-10T06:39:01.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T06:39:01.617 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:01 smithi181 ceph-mon[36721]: from='osd.4 [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:39:01.617 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:01 smithi181 ceph-mon[36721]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:39:01.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:01 smithi073 ceph-mon[29063]: from='osd.4 [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:39:01.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:01 smithi073 ceph-mon[29063]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:39:01.879 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: osdmap e33: 5 total, 4 up, 5 in 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: from='osd.4 [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:02.591 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:02.592 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-4[49219]: 2024-02-10T06:39:02.497+0000 7fd7a888e700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T06:39:02.592 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 10 06:39:02 smithi181 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-4[49219]: 2024-02-10T06:39:02.504+0000 7fd7a2687700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: osdmap e33: 5 total, 4 up, 5 in 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: from='osd.4 [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:02.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:03.717 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T06:39:03.718 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T06:39:03.718 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-10T06:39:03.718 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-10T06:39:03.718 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124461 s, 842 MB/s 2024-02-10T06:39:03.718 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-10T06:39:03.741 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:03 smithi181 ceph-mon[36721]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-10T06:39:03.741 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:03 smithi181 ceph-mon[36721]: osdmap e34: 5 total, 4 up, 5 in 2024-02-10T06:39:03.741 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:03 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:03.741 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:03 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:03.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:03 smithi073 ceph-mon[29063]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-10T06:39:03.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:03 smithi073 ceph-mon[29063]: osdmap e34: 5 total, 4 up, 5 in 2024-02-10T06:39:03.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:03.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:04.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: purged_snaps scrub starts 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: purged_snaps scrub ok 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: osd.4 [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] boot 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T06:39:04.529 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:04.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: purged_snaps scrub starts 2024-02-10T06:39:04.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: purged_snaps scrub ok 2024-02-10T06:39:04.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:39:04.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:04.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: osd.4 [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] boot 2024-02-10T06:39:04.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T06:39:04.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:39:05.502 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:39:05.762 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:05 smithi181 ceph-mon[36721]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T06:39:05.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:05 smithi073 ceph-mon[29063]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T06:39:06.739 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:06 smithi181 ceph-mon[36721]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:06.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:06 smithi073 ceph-mon[29063]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:07.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.626+0000 7f7bab09f700 1 -- 172.21.15.181:0/2033605483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 msgr2=0x7f7ba40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:07.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.626+0000 7f7bab09f700 1 --2- 172.21.15.181:0/2033605483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40fe550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b90004970 tx=0x7f7b90009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:07.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- 172.21.15.181:0/2033605483 shutdown_connections 2024-02-10T06:39:07.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 --2- 172.21.15.181:0/2033605483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40fe550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:07.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 --2- 172.21.15.181:0/2033605483 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7ba40fcef0 0x7f7ba40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- 172.21.15.181:0/2033605483 >> 172.21.15.181:0/2033605483 conn(0x7f7ba40f8620 msgr2=0x7f7ba40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- 172.21.15.181:0/2033605483 shutdown_connections 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- 172.21.15.181:0/2033605483 wait complete. 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 Processor -- start 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- start start 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7ba40fcef0 0x7f7ba4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba4070970 con 0x7f7ba40fcef0 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7bab09f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba4070ae0 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.627+0000 7f7ba3fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba3fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:33878/0 (socket says 172.21.15.181:33878) 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba3fff700 1 -- 172.21.15.181:0/3529921833 learned_addr learned my addr 172.21.15.181:0/3529921833 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba8e3b700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7ba40fcef0 0x7f7ba4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba3fff700 1 -- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7ba40fcef0 msgr2=0x7f7ba4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:07.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba3fff700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7ba40fcef0 0x7f7ba4071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba3fff700 1 -- 172.21.15.181:0/3529921833 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b90005040 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba3fff700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40702e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000b370 tx=0x7f7b90009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9000e040 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b9000bd60 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba4070d60 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9001c630 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.628+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba41159c0 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.629+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba404bed0 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.630+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7b90013a80 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.630+0000 7f7ba17fa700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7b940692a0 0x7f7b9406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.630+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f7b90088090 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.630+0000 7f7ba8e3b700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7b940692a0 0x7f7b9406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:07.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.631+0000 7f7ba8e3b700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7b940692a0 0x7f7b9406b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006fd0 tx=0x7f7b98008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:07.636 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.634+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7b90058340 con 0x7f7ba40fe0e0 2024-02-10T06:39:07.829 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:07.828+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ba4102ac0 con 0x7f7b940692a0 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: Detected new or changed devices on smithi181 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: Adjusting osd_memory_target on smithi181 to 7968M 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:08.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:08.093 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:08.093 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: Detected new or changed devices on smithi181 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: Adjusting osd_memory_target on smithi181 to 7968M 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:08.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:08.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:08.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:08.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:08.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:09.034 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:08 smithi181 ceph-mon[36721]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:09.035 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:08 smithi181 ceph-mon[36721]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:39:09.035 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:39:09.035 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:39:09.035 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:08 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:09.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:08 smithi073 ceph-mon[29063]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:09.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:08 smithi073 ceph-mon[29063]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:39:09.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:39:09.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:39:09.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:08 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/1110227838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd38e72-1e2b-41a3-974c-c5d431919187"}]: dispatch 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd38e72-1e2b-41a3-974c-c5d431919187"}]: dispatch 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bd38e72-1e2b-41a3-974c-c5d431919187"}]': finished 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:11.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:10 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/4246818492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/1110227838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd38e72-1e2b-41a3-974c-c5d431919187"}]: dispatch 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bd38e72-1e2b-41a3-974c-c5d431919187"}]: dispatch 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bd38e72-1e2b-41a3-974c-c5d431919187"}]': finished 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:11.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:10 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/4246818492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:39:13.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:12 smithi181 ceph-mon[36721]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:13.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:13.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:12 smithi073 ceph-mon[29063]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:13.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:15.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:14 smithi181 ceph-mon[36721]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:15.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:14 smithi073 ceph-mon[29063]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:17.013 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:16 smithi181 ceph-mon[36721]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:16 smithi073 ceph-mon[29063]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:18.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:39:18.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:17 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:39:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:17 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:18.941 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:18 smithi181 ceph-mon[36721]: Deploying daemon osd.5 on smithi181 2024-02-10T06:39:18.941 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:18 smithi181 ceph-mon[36721]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:19.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:18 smithi073 ceph-mon[29063]: Deploying daemon osd.5 on smithi181 2024-02-10T06:39:19.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:18 smithi073 ceph-mon[29063]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:21.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:20 smithi181 ceph-mon[36721]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:20 smithi073 ceph-mon[29063]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:23.246 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:22 smithi181 ceph-mon[36721]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:23.247 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:23.247 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:23.247 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:22 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:23.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:22 smithi073 ceph-mon[29063]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:23.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:23.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:23.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:22 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:25.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:24 smithi181 ceph-mon[36721]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:25.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:24 smithi073 ceph-mon[29063]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:25.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.544+0000 7f7ba17fa700 1 -- 172.21.15.181:0/3529921833 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7ba4102ac0 con 0x7f7b940692a0 2024-02-10T06:39:25.545 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7b940692a0 msgr2=0x7f7b9406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7b940692a0 0x7f7b9406b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006fd0 tx=0x7f7b98008040 comp rx=0 tx=0).stop 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 msgr2=0x7f7ba40702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40702e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000b370 tx=0x7f7b90009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 shutdown_connections 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7b940692a0 0x7f7b9406b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7ba40fcef0 0x7f7ba4071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 --2- 172.21.15.181:0/3529921833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ba40fe0e0 0x7f7ba40702e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 >> 172.21.15.181:0/3529921833 conn(0x7f7ba40f8620 msgr2=0x7f7ba41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 shutdown_connections 2024-02-10T06:39:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:25.546+0000 7f7bab09f700 1 -- 172.21.15.181:0/3529921833 wait complete. 2024-02-10T06:39:26.082 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T06:39:26.125 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-10T06:39:26.125 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T06:39:26.395 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:26.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:26 smithi073 ceph-mon[29063]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:26.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:26 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:26.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:26 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:26.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:26 smithi073 ceph-mon[29063]: from='osd.5 [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:39:26.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:26 smithi073 ceph-mon[29063]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:39:26.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:26 smithi181 ceph-mon[36721]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:26.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:26 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:26.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:26 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:26.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:26 smithi181 ceph-mon[36721]: from='osd.5 [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:39:26.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:26 smithi181 ceph-mon[36721]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:39:27.766 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:39:27.766 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T06:39:27.766 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:27.767 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='osd.5 [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:27.767 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:27.767 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:27.767 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:27.767 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:27.767 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-5[56128]: 2024-02-10T06:39:27.528+0000 7f1aba64c700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T06:39:27.767 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 10 06:39:27 smithi181 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-5[56128]: 2024-02-10T06:39:27.536+0000 7f1ab5447700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:39:27.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:39:27.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T06:39:27.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:27.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='osd.5 [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:27.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-10T06:39:27.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:27.783 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:27.783 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 -- 172.21.15.73:0/1627921934 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 msgr2=0x7f1ca8102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/1627921934 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca8102520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4004940 tx=0x7f1ca4009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 -- 172.21.15.73:0/1627921934 shutdown_connections 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/1627921934 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca8102520 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/1627921934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca8066c50 0x7f1ca80ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 -- 172.21.15.73:0/1627921934 >> 172.21.15.73:0/1627921934 conn(0x7f1ca80f7520 msgr2=0x7f1ca80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 -- 172.21.15.73:0/1627921934 shutdown_connections 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.164+0000 7f1cb07fe700 1 -- 172.21.15.73:0/1627921934 wait complete. 2024-02-10T06:39:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cb07fe700 1 Processor -- start 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cb07fe700 1 -- start start 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cb07fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca8066c50 0x7f1ca810d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cb07fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca810de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cb07fe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca810e490 con 0x7f1ca81000e0 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cb07fe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca810ed30 con 0x7f1ca8066c50 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cadd99700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca810de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cadd99700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca810de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47814/0 (socket says 172.21.15.73:47814) 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cadd99700 1 -- 172.21.15.73:0/4063851659 learned_addr learned my addr 172.21.15.73:0/4063851659 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cae59a700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca8066c50 0x7f1ca810d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cadd99700 1 -- 172.21.15.73:0/4063851659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca8066c50 msgr2=0x7f1ca810d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cadd99700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca8066c50 0x7f1ca810d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.165+0000 7f1cadd99700 1 -- 172.21.15.73:0/4063851659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4005040 con 0x7f1ca81000e0 2024-02-10T06:39:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.166+0000 7f1cadd99700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca810de90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009360 tx=0x7f1ca4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.166+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca400e040 con 0x7f1ca81000e0 2024-02-10T06:39:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.166+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ca4004290 con 0x7f1ca81000e0 2024-02-10T06:39:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.166+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca810efb0 con 0x7f1ca81000e0 2024-02-10T06:39:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.166+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca401c630 con 0x7f1ca81000e0 2024-02-10T06:39:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.166+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca810f4f0 con 0x7f1ca81000e0 2024-02-10T06:39:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.167+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca804bed0 con 0x7f1ca81000e0 2024-02-10T06:39:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.167+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1ca4013980 con 0x7f1ca81000e0 2024-02-10T06:39:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.168+0000 7f1c9effd700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c940692b0 0x7f1c9406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.168+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f1ca4087f60 con 0x7f1ca81000e0 2024-02-10T06:39:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.168+0000 7f1cae59a700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c940692b0 0x7f1c9406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.168+0000 7f1cae59a700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c940692b0 0x7f1c9406b760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1c980041d0 tx=0x7f1c980043c0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:28.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.172+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1ca40b9bd0 con 0x7f1ca81000e0 2024-02-10T06:39:28.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.361+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f1ca8105a10 con 0x7f1ca81000e0 2024-02-10T06:39:28.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.362+0000 7f1c9effd700 1 -- 172.21.15.73:0/4063851659 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f1ca4058210 con 0x7f1ca81000e0 2024-02-10T06:39:28.363 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:39:28.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c940692b0 msgr2=0x7f1c9406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:28.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c940692b0 0x7f1c9406b760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1c980041d0 tx=0x7f1c980043c0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 msgr2=0x7f1ca810de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:28.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca810de90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009360 tx=0x7f1ca4009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 shutdown_connections 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c940692b0 0x7f1c9406b760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca8066c50 0x7f1ca810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 --2- 172.21.15.73:0/4063851659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ca81000e0 0x7f1ca810de90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 >> 172.21.15.73:0/4063851659 conn(0x7f1ca80f7520 msgr2=0x7f1ca80f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 shutdown_connections 2024-02-10T06:39:28.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:28.364+0000 7f1cb07fe700 1 -- 172.21.15.73:0/4063851659 wait complete. 2024-02-10T06:39:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:28 smithi073 ceph-mon[29063]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:28 smithi073 ceph-mon[29063]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-10T06:39:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:28 smithi073 ceph-mon[29063]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T06:39:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:28 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:28 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:28.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:28 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/4063851659' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:39:28.675 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:28 smithi181 ceph-mon[36721]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:28.675 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:28 smithi181 ceph-mon[36721]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-10T06:39:28.675 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:28 smithi181 ceph-mon[36721]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T06:39:28.675 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:28 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:28.675 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:28 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:28.675 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:28 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/4063851659' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:39:29.137 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1707547143,"num_in_osds":6,"osd_in_since":1707547150,"num_remapped_pgs":0} 2024-02-10T06:39:29.740 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:29 smithi181 ceph-mon[36721]: purged_snaps scrub starts 2024-02-10T06:39:29.740 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:29 smithi181 ceph-mon[36721]: purged_snaps scrub ok 2024-02-10T06:39:29.740 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:29 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:29.740 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:29 smithi181 ceph-mon[36721]: osd.5 [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] boot 2024-02-10T06:39:29.741 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:29 smithi181 ceph-mon[36721]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T06:39:29.741 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:29 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:29 smithi073 ceph-mon[29063]: purged_snaps scrub starts 2024-02-10T06:39:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:29 smithi073 ceph-mon[29063]: purged_snaps scrub ok 2024-02-10T06:39:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:29 smithi073 ceph-mon[29063]: osd.5 [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] boot 2024-02-10T06:39:29.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:29 smithi073 ceph-mon[29063]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T06:39:29.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:29 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:39:30.138 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T06:39:30.408 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:30.842 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:30.842 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.842 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:39:30.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:30.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:30.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:30.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:31.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:31 smithi073 ceph-mon[29063]: Detected new or changed devices on smithi181 2024-02-10T06:39:31.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:31 smithi073 ceph-mon[29063]: Adjusting osd_memory_target on smithi181 to 5312M 2024-02-10T06:39:31.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:31 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:31.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:31 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:31.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:31 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:31.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:31 smithi181 ceph-mon[36721]: Detected new or changed devices on smithi181 2024-02-10T06:39:31.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:31 smithi181 ceph-mon[36721]: Adjusting osd_memory_target on smithi181 to 5312M 2024-02-10T06:39:31.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:31 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:31.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:31 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:31.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:31 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:39:32.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- 172.21.15.73:0/913437103 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2858066bf0 msgr2=0x7f28580ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 --2- 172.21.15.73:0/913437103 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2858066bf0 0x7f28580ffb80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2854004940 tx=0x7f2854009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- 172.21.15.73:0/913437103 shutdown_connections 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 --2- 172.21.15.73:0/913437103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28581000c0 0x7f28581024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 --2- 172.21.15.73:0/913437103 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2858066bf0 0x7f28580ffb80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- 172.21.15.73:0/913437103 >> 172.21.15.73:0/913437103 conn(0x7f28580f7590 msgr2=0x7f28580f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- 172.21.15.73:0/913437103 shutdown_connections 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- 172.21.15.73:0/913437103 wait complete. 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 Processor -- start 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- start start 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2858066bf0 0x7f28580fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 0x7f28580fba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:32.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580fc110 con 0x7f28581000c0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.128+0000 7f2860654700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580fc280 con 0x7f2858066bf0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 0x7f28580fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 0x7f28580fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58406/0 (socket says 172.21.15.73:58406) 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 -- 172.21.15.73:0/2064074375 learned_addr learned my addr 172.21.15.73:0/2064074375 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285e3f0700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2858066bf0 0x7f28580fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 -- 172.21.15.73:0/2064074375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2858066bf0 msgr2=0x7f28580fd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2858066bf0 0x7f28580fd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 -- 172.21.15.73:0/2064074375 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854005040 con 0x7f28581000c0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285e3f0700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2858066bf0 0x7f28580fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f285dbef700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 0x7f28580fba80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f284800b790 tx=0x7f2848009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284800f770 con 0x7f28581000c0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f284800fda0 con 0x7f28581000c0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284800d310 con 0x7f28581000c0 2024-02-10T06:39:32.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28580fc560 con 0x7f28581000c0 2024-02-10T06:39:32.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.129+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28580fc8e0 con 0x7f28581000c0 2024-02-10T06:39:32.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.130+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285804bed0 con 0x7f28581000c0 2024-02-10T06:39:32.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.131+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2848010420 con 0x7f28581000c0 2024-02-10T06:39:32.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.131+0000 7f284effd700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f28440692a0 0x7f284406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:32.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.131+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2848086e70 con 0x7f28581000c0 2024-02-10T06:39:32.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.131+0000 7f285e3f0700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f28440692a0 0x7f284406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:32.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.131+0000 7f285e3f0700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f28440692a0 0x7f284406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f285400b3b0 tx=0x7f2854009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:32.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.135+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2848056f70 con 0x7f28581000c0 2024-02-10T06:39:32.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.320+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2858115e90 con 0x7f28581000c0 2024-02-10T06:39:32.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.321+0000 7f284effd700 1 -- 172.21.15.73:0/2064074375 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2848056b00 con 0x7f28581000c0 2024-02-10T06:39:32.322 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:39:32.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f28440692a0 msgr2=0x7f284406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f28440692a0 0x7f284406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f285400b3b0 tx=0x7f2854009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 msgr2=0x7f28580fba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 0x7f28580fba80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f284800b790 tx=0x7f2848009250 comp rx=0 tx=0).stop 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 shutdown_connections 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f28440692a0 0x7f284406b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2858066bf0 0x7f28580fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 --2- 172.21.15.73:0/2064074375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28581000c0 0x7f28580fba80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.323+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 >> 172.21.15.73:0/2064074375 conn(0x7f28580f7590 msgr2=0x7f28580f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.324+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 shutdown_connections 2024-02-10T06:39:32.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:32.324+0000 7f2860654700 1 -- 172.21.15.73:0/2064074375 wait complete. 2024-02-10T06:39:33.047 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:32 smithi073 ceph-mon[29063]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:33.047 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:32 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2064074375' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:39:33.078 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1707547168,"num_in_osds":6,"osd_in_since":1707547150,"num_remapped_pgs":0} 2024-02-10T06:39:33.079 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:39:33.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:32 smithi181 ceph-mon[36721]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:33.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:32 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2064074375' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:39:33.341 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:35.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:34 smithi073 ceph-mon[29063]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 -- 172.21.15.73:0/1103872803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fdf60 msgr2=0x7faf880fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/1103872803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fdf60 0x7faf880fe330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf7c004940 tx=0x7faf7c009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 -- 172.21.15.73:0/1103872803 shutdown_connections 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/1103872803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf880fe900 0x7faf88106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/1103872803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fdf60 0x7faf880fe330 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 -- 172.21.15.73:0/1103872803 >> 172.21.15.73:0/1103872803 conn(0x7faf880f7a10 msgr2=0x7faf880f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 -- 172.21.15.73:0/1103872803 shutdown_connections 2024-02-10T06:39:35.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.165+0000 7faf8d7f6700 1 -- 172.21.15.73:0/1103872803 wait complete. 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf8d7f6700 1 Processor -- start 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf8d7f6700 1 -- start start 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf8d7f6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf880fdf60 0x7faf88115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf8d7f6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 0x7faf88116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf8d7f6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf881168f0 con 0x7faf880fe900 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf8d7f6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8810fb70 con 0x7faf880fdf60 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf867fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 0x7faf88116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf867fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 0x7faf88116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58426/0 (socket says 172.21.15.73:58426) 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf867fc700 1 -- 172.21.15.73:0/2614656089 learned_addr learned my addr 172.21.15.73:0/2614656089 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:35.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf86ffd700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf880fdf60 0x7faf88115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf867fc700 1 -- 172.21.15.73:0/2614656089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf880fdf60 msgr2=0x7faf88115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf867fc700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf880fdf60 0x7faf88115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.166+0000 7faf867fc700 1 -- 172.21.15.73:0/2614656089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c005040 con 0x7faf880fe900 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.167+0000 7faf867fc700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 0x7faf88116210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b790 tx=0x7faf78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.167+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf7800f6f0 con 0x7faf880fe900 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.167+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8810fdf0 con 0x7faf880fe900 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.167+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf7800fd20 con 0x7faf880fe900 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.167+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf780103a0 con 0x7faf880fe900 2024-02-10T06:39:35.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.167+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf88110330 con 0x7faf880fe900 2024-02-10T06:39:35.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.168+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf881069b0 con 0x7faf880fe900 2024-02-10T06:39:35.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.168+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faf78011400 con 0x7faf880fe900 2024-02-10T06:39:35.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.169+0000 7faf6f7fe700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7faf700692a0 0x7faf7006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:35.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.169+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faf78086c30 con 0x7faf880fe900 2024-02-10T06:39:35.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.169+0000 7faf86ffd700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7faf700692a0 0x7faf7006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:35.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.169+0000 7faf86ffd700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7faf700692a0 0x7faf7006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf7c004560 tx=0x7faf7c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:35.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.173+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7faf78056d30 con 0x7faf880fe900 2024-02-10T06:39:35.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:34 smithi181 ceph-mon[36721]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:35.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.357+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf8804bed0 con 0x7faf880fe900 2024-02-10T06:39:35.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.358+0000 7faf6f7fe700 1 -- 172.21.15.73:0/2614656089 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11070 (secure 0 0 0) 0x7faf780568c0 con 0x7faf880fe900 2024-02-10T06:39:35.359 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:39:35.359 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":41,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:33:49.428986+0000","modified":"2024-02-10T06:39:29.531882+0000","last_up_change":"2024-02-10T06:39:28.526237+0000","last_in_change":"2024-02-10T06:39:10.116034+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:38:17.748559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8e333d6f-68ed-41e7-a417-bed0ad6e2ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6803","nonce":1003470321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6805","nonce":1003470321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6809","nonce":1003470321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6807","nonce":1003470321}]},"public_addr":"172.21.15.73:6803/1003470321","cluster_addr":"172.21.15.73:6805/1003470321","heartbeat_back_addr":"172.21.15.73:6809/1003470321","heartbeat_front_addr":"172.21.15.73:6807/1003470321","state":["exists","up"]},{"osd":1,"uuid":"dedbb82b-7b4d-47eb-8621-2a35fa20a05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6811","nonce":2186552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6813","nonce":2186552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6817","nonce":2186552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6815","nonce":2186552820}]},"public_addr":"172.21.15.73:6811/2186552820","cluster_addr":"172.21.15.73:6813/2186552820","heartbeat_back_addr":"172.21.15.73:6817/2186552820","heartbeat_front_addr":"172.21.15.73:6815/2186552820","state":["exists","up"]},{"osd":2,"uuid":"6e7d61ae-a65b-43db-ac2d-aec724834f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6819","nonce":4245951128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6821","nonce":4245951128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6825","nonce":4245951128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6823","nonce":4245951128}]},"public_addr":"172.21.15.73:6819/4245951128","cluster_addr":"172.21.15.73:6821/4245951128","heartbeat_back_addr":"172.21.15.73:6825/4245951128","heartbeat_front_addr":"172.21.15.73:6823/4245951128","state":["exists","up"]},{"osd":3,"uuid":"2f17b271-99d5-4b38-89fb-175414c59351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6801","nonce":156965919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6803","nonce":156965919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6807","nonce":156965919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6805","nonce":156965919}]},"public_addr":"172.21.15.181:6801/156965919","cluster_addr":"172.21.15.181:6803/156965919","heartbeat_back_addr":"172.21.15.181:6807/156965919","heartbeat_front_addr":"172.21.15.181:6805/156965919","state":["exists","up"]},{"osd":4,"uuid":"28fd87c6-5f23-44af-87a6-d3cf81095f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6809","nonce":849727482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6811","nonce":849727482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6815","nonce":849727482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6813","nonce":849727482}]},"public_addr":"172.21.15.181:6809/849727482","cluster_addr":"172.21.15.181:6811/849727482","heartbeat_back_addr":"172.21.15.181:6815/849727482","heartbeat_front_addr":"172.21.15.181:6813/849727482","state":["exists","up"]},{"osd":5,"uuid":"9bd38e72-1e2b-41a3-974c-c5d431919187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6817","nonce":1157810956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6819","nonce":1157810956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6823","nonce":1157810956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6821","nonce":1157810956}]},"public_addr":"172.21.15.181:6817/1157810956","cluster_addr":"172.21.15.181:6819/1157810956","heartbeat_back_addr":"172.21.15.181:6823/1157810956","heartbeat_front_addr":"172.21.15.181:6821/1157810956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:14.313049+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:44.358563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:14.291417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:37.782333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:02.393199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:26.947433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:0/3920786161":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/4190786023":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6800/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/1749918597":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6801/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6801/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/3119085620":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/3298620834":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/1695656680":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/2932627761":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6800/194898351":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6800/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/1643478797":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/810500280":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6801/194898351":"2024-02-11T06:34:17.719875+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7faf700692a0 msgr2=0x7faf7006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7faf700692a0 0x7faf7006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf7c004560 tx=0x7faf7c009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 msgr2=0x7faf88116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 0x7faf88116210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b790 tx=0x7faf78009250 comp rx=0 tx=0).stop 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 shutdown_connections 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7faf700692a0 0x7faf7006b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf880fdf60 0x7faf88115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 --2- 172.21.15.73:0/2614656089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf880fe900 0x7faf88116210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 >> 172.21.15.73:0/2614656089 conn(0x7faf880f7a10 msgr2=0x7faf880fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 shutdown_connections 2024-02-10T06:39:35.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:35.360+0000 7faf8d7f6700 1 -- 172.21.15.73:0/2614656089 wait complete. 2024-02-10T06:39:35.994 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:35 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2614656089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:39:36.019 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T06:38:17.748559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-10T06:39:36.019 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T06:39:36.270 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:36.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:35 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2614656089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:39:37.093 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:36 smithi073 ceph-mon[29063]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:37.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:36 smithi181 ceph-mon[36721]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:38.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 -- 172.21.15.73:0/3667076699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fdf70 msgr2=0x7f85f40fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:38.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3667076699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fdf70 0x7f85f40fe340 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f85e4004940 tx=0x7f85e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:38.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 -- 172.21.15.73:0/3667076699 shutdown_connections 2024-02-10T06:39:38.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3667076699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f40fe880 0x7f85f4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3667076699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fdf70 0x7f85f40fe340 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 -- 172.21.15.73:0/3667076699 >> 172.21.15.73:0/3667076699 conn(0x7f85f40f79d0 msgr2=0x7f85f40f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 -- 172.21.15.73:0/3667076699 shutdown_connections 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.055+0000 7f85fa78e700 1 -- 172.21.15.73:0/3667076699 wait complete. 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85fa78e700 1 Processor -- start 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85fa78e700 1 -- start start 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85fa78e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f40fdf70 0x7f85f4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85fa78e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 0x7f85f4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85fa78e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4114710 con 0x7f85f40fe880 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85fa78e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f410d9b0 con 0x7f85f40fdf70 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f37fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 0x7f85f4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f37fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 0x7f85f4114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58438/0 (socket says 172.21.15.73:58438) 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f37fe700 1 -- 172.21.15.73:0/3520525840 learned_addr learned my addr 172.21.15.73:0/3520525840 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:38.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f3fff700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f40fdf70 0x7f85f4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f37fe700 1 -- 172.21.15.73:0/3520525840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f40fdf70 msgr2=0x7f85f4113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f37fe700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f40fdf70 0x7f85f4113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.056+0000 7f85f37fe700 1 -- 172.21.15.73:0/3520525840 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e4005040 con 0x7f85f40fe880 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.057+0000 7f85f37fe700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 0x7f85f4114030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f85e800b790 tx=0x7f85e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.057+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85e800f770 con 0x7f85f40fe880 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.057+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85e800fda0 con 0x7f85f40fe880 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.057+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f410dc90 con 0x7f85f40fe880 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.057+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85e800d310 con 0x7f85f40fe880 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.057+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f410e1d0 con 0x7f85f40fe880 2024-02-10T06:39:38.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.058+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f4104670 con 0x7f85f40fe880 2024-02-10T06:39:38.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.058+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f85e8010420 con 0x7f85f40fe880 2024-02-10T06:39:38.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.059+0000 7f85f0ff9700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f85dc0692b0 0x7f85dc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:38.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.059+0000 7f85f3fff700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f85dc0692b0 0x7f85dc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:38.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.059+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f85e8086de0 con 0x7f85f40fe880 2024-02-10T06:39:38.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.059+0000 7f85f3fff700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f85dc0692b0 0x7f85dc06b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f85e400b3b0 tx=0x7f85e4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:38.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.063+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f85e8056f00 con 0x7f85f40fe880 2024-02-10T06:39:38.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.244+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f85f410a0b0 con 0x7f85f40fe880 2024-02-10T06:39:38.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.245+0000 7f85f0ff9700 1 -- 172.21.15.73:0/3520525840 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85e8056a90 con 0x7f85f40fe880 2024-02-10T06:39:38.246 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f85dc0692b0 msgr2=0x7f85dc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f85dc0692b0 0x7f85dc06b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f85e400b3b0 tx=0x7f85e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 msgr2=0x7f85f4114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 0x7f85f4114030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f85e800b790 tx=0x7f85e8009250 comp rx=0 tx=0).stop 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 shutdown_connections 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f85dc0692b0 0x7f85dc06b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f40fdf70 0x7f85f4113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 --2- 172.21.15.73:0/3520525840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85f40fe880 0x7f85f4114030 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 >> 172.21.15.73:0/3520525840 conn(0x7f85f40f79d0 msgr2=0x7f85f41017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 shutdown_connections 2024-02-10T06:39:38.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:38.248+0000 7f85fa78e700 1 -- 172.21.15.73:0/3520525840 wait complete. 2024-02-10T06:39:38.902 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T06:39:38.902 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T06:39:39.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:38 smithi073 ceph-mon[29063]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:39.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:38 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3520525840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:39:39.156 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:39.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:38 smithi181 ceph-mon[36721]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:39.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:38 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3520525840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:39:40.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 -- 172.21.15.73:0/3626540074 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fc4b0 msgr2=0x7fcb3c0fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/3626540074 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c0fc8c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009ce0 tx=0x7fcb34004d10 comp rx=0 tx=0).stop 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 -- 172.21.15.73:0/3626540074 shutdown_connections 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/3626540074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c0fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/3626540074 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c0fc8c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 -- 172.21.15.73:0/3626540074 >> 172.21.15.73:0/3626540074 conn(0x7fcb3c0f6d20 msgr2=0x7fcb3c0f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 -- 172.21.15.73:0/3626540074 shutdown_connections 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 -- 172.21.15.73:0/3626540074 wait complete. 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.860+0000 7fcb41ab3700 1 Processor -- start 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb41ab3700 1 -- start start 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb41ab3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c110b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb41ab3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb41ab3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0fbf80 con 0x7fcb3c0fce00 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb41ab3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0fc0c0 con 0x7fcb3c0fc4b0 2024-02-10T06:39:40.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c113050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57964/0 (socket says 172.21.15.73:57964) 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 -- 172.21.15.73:0/2100849940 learned_addr learned my addr 172.21.15.73:0/2100849940 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3b7fe700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c110b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 -- 172.21.15.73:0/2100849940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fc4b0 msgr2=0x7fcb3c110b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c110b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 -- 172.21.15.73:0/2100849940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb34009940 con 0x7fcb3c0fce00 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3b7fe700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c110b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb3affd700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c113050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000b790 tx=0x7fcb30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3000f770 con 0x7fcb3c0fce00 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb3000fda0 con 0x7fcb3c0fce00 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.861+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c1136e0 con 0x7fcb3c0fce00 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.862+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3000d310 con 0x7fcb3c0fce00 2024-02-10T06:39:40.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.862+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c113c20 con 0x7fcb3c0fce00 2024-02-10T06:39:40.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.862+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c106a00 con 0x7fcb3c0fce00 2024-02-10T06:39:40.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.863+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcb30010420 con 0x7fcb3c0fce00 2024-02-10T06:39:40.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.863+0000 7fcb23fff700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcb240692a0 0x7fcb2406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:40.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.863+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcb30086e70 con 0x7fcb3c0fce00 2024-02-10T06:39:40.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.863+0000 7fcb3b7fe700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcb240692a0 0x7fcb2406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:40.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.864+0000 7fcb3b7fe700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcb240692a0 0x7fcb2406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009470 tx=0x7fcb34004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:40.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:40.870+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcb300534e0 con 0x7fcb3c0fce00 2024-02-10T06:39:41.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:40 smithi073 ceph-mon[29063]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:41.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.122+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fcb3c104ac0 con 0x7fcb3c0fce00 2024-02-10T06:39:41.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.126+0000 7fcb23fff700 1 -- 172.21.15.73:0/2100849940 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fcb30056b00 con 0x7fcb3c0fce00 2024-02-10T06:39:41.127 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2024-02-10T06:39:41.127 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQAtGsdlrAxhBxAAD/KPzDEj1eCqQlBHs1U9yg== 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcb240692a0 msgr2=0x7fcb2406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcb240692a0 0x7fcb2406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009470 tx=0x7fcb34004d10 comp rx=0 tx=0).stop 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 msgr2=0x7fcb3c113050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c113050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000b790 tx=0x7fcb30009250 comp rx=0 tx=0).stop 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 shutdown_connections 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcb240692a0 0x7fcb2406b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb3c0fc4b0 0x7fcb3c110b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 --2- 172.21.15.73:0/2100849940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb3c0fce00 0x7fcb3c113050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 >> 172.21.15.73:0/2100849940 conn(0x7fcb3c0f6d20 msgr2=0x7fcb3c1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 shutdown_connections 2024-02-10T06:39:41.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:41.128+0000 7fcb41ab3700 1 -- 172.21.15.73:0/2100849940 wait complete. 2024-02-10T06:39:41.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:40 smithi181 ceph-mon[36721]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:41.807 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:39:41.808 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T06:39:41.808 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T06:39:41.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T06:39:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:41 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2100849940' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:39:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:41 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2100849940' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:39:42.123 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi181/config 2024-02-10T06:39:42.232 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:41 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2100849940' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:39:42.232 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:41 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2100849940' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:39:43.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:42 smithi073 ceph-mon[29063]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:43.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:43.222 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:42 smithi181 ceph-mon[36721]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:43.222 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:43.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 -- 172.21.15.181:0/2620952394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fe2b0 msgr2=0x7fab840fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 --2- 172.21.15.181:0/2620952394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fe2b0 0x7fab840fe700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fab74004940 tx=0x7fab74009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 -- 172.21.15.181:0/2620952394 shutdown_connections 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 --2- 172.21.15.181:0/2620952394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fe2b0 0x7fab840fe700 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 --2- 172.21.15.181:0/2620952394 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab840fb220 0x7fab840fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 -- 172.21.15.181:0/2620952394 >> 172.21.15.181:0/2620952394 conn(0x7fab840f6ca0 msgr2=0x7fab840f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.487+0000 7fab88995700 1 -- 172.21.15.181:0/2620952394 shutdown_connections 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 -- 172.21.15.181:0/2620952394 wait complete. 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 Processor -- start 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 -- start start 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 0x7fab840fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab840fe2b0 0x7fab840fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840fc7c0 con 0x7fab840fe2b0 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab88995700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840fc930 con 0x7fab840fb220 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab8259c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 0x7fab840fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab8259c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 0x7fab840fdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55416/0 (socket says 172.21.15.181:55416) 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.488+0000 7fab8259c700 1 -- 172.21.15.181:0/3371893892 learned_addr learned my addr 172.21.15.181:0/3371893892 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab81d9b700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab840fe2b0 0x7fab840fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab8259c700 1 -- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab840fe2b0 msgr2=0x7fab840fc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab8259c700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab840fe2b0 0x7fab840fc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab8259c700 1 -- 172.21.15.181:0/3371893892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab74005040 con 0x7fab840fb220 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab8259c700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 0x7fab840fdd40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c8b0 tx=0x7fab6c004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c00a760 con 0x7fab840fb220 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab840fcc10 con 0x7fab840fb220 2024-02-10T06:39:43.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab6c00ad90 con 0x7fab840fb220 2024-02-10T06:39:43.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c0113a0 con 0x7fab840fb220 2024-02-10T06:39:43.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.489+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab84115a10 con 0x7fab840fb220 2024-02-10T06:39:43.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.490+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8404bed0 con 0x7fab840fb220 2024-02-10T06:39:43.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.491+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fab6c012400 con 0x7fab840fb220 2024-02-10T06:39:43.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.491+0000 7fab7affd700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fab700692a0 0x7fab7006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:43.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.491+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fab6c086e90 con 0x7fab840fb220 2024-02-10T06:39:43.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.491+0000 7fab81d9b700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fab700692a0 0x7fab7006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:43.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.492+0000 7fab81d9b700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fab700692a0 0x7fab7006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fab74004f90 tx=0x7fab74009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:43.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.496+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fab6c056f90 con 0x7fab840fb220 2024-02-10T06:39:43.754 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.753+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fab84105a40 con 0x7fab840fb220 2024-02-10T06:39:43.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.758+0000 7fab7affd700 1 -- 172.21.15.181:0/3371893892 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fab6c056b20 con 0x7fab840fb220 2024-02-10T06:39:43.759 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-02-10T06:39:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAvGsdl+/YDLRAAfFy5mwn90DTDS+CuVThHRg== 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fab700692a0 msgr2=0x7fab7006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fab700692a0 0x7fab7006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fab74004f90 tx=0x7fab74009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 msgr2=0x7fab840fdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 0x7fab840fdd40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c8b0 tx=0x7fab6c004470 comp rx=0 tx=0).stop 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 shutdown_connections 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fab700692a0 0x7fab7006b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab840fb220 0x7fab840fdd40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 --2- 172.21.15.181:0/3371893892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab840fe2b0 0x7fab840fc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:43.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 >> 172.21.15.181:0/3371893892 conn(0x7fab840f6ca0 msgr2=0x7fab841014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:43.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 shutdown_connections 2024-02-10T06:39:43.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-10T06:39:43.760+0000 7fab88995700 1 -- 172.21.15.181:0/3371893892 wait complete. 2024-02-10T06:39:44.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:43 smithi181 ceph-mon[36721]: from='client.? 172.21.15.181:0/3371893892' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:39:44.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:43 smithi181 ceph-mon[36721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:39:44.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:43 smithi181 ceph-mon[36721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:39:44.326 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:39:44.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T06:39:44.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T06:39:44.369 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T06:39:44.369 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T06:39:44.369 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T06:39:44.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:43 smithi073 ceph-mon[29063]: from='client.? 172.21.15.181:0/3371893892' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:39:44.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:43 smithi073 ceph-mon[29063]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:39:44.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:43 smithi073 ceph-mon[29063]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:39:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:45.070 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:44 smithi073 ceph-mon[29063]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:45.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:44 smithi181 ceph-mon[36721]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 -- 172.21.15.73:0/1140503788 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48581038f0 msgr2=0x7f48580fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 --2- 172.21.15.73:0/1140503788 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48581038f0 0x7f48580fcf60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4848004940 tx=0x7f4848009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 -- 172.21.15.73:0/1140503788 shutdown_connections 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 --2- 172.21.15.73:0/1140503788 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48581038f0 0x7f48580fcf60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 --2- 172.21.15.73:0/1140503788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4858102f50 0x7f4858103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 -- 172.21.15.73:0/1140503788 >> 172.21.15.73:0/1140503788 conn(0x7f48580f8ae0 msgr2=0x7f48580faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 -- 172.21.15.73:0/1140503788 shutdown_connections 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.310+0000 7f485ea75700 1 -- 172.21.15.73:0/1140503788 wait complete. 2024-02-10T06:39:46.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485ea75700 1 Processor -- start 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485ea75700 1 -- start start 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485ea75700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 0x7f4858113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485ea75700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48581038f0 0x7f4858113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485ea75700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4858114620 con 0x7f4858102f50 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485ea75700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485810d950 con 0x7f48581038f0 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 0x7f4858113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 0x7f4858113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57976/0 (socket says 172.21.15.73:57976) 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 -- 172.21.15.73:0/946590496 learned_addr learned my addr 172.21.15.73:0/946590496 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f4857fff700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48581038f0 0x7f4858113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 -- 172.21.15.73:0/946590496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48581038f0 msgr2=0x7f4858113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48581038f0 0x7f4858113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 -- 172.21.15.73:0/946590496 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848005040 con 0x7f4858102f50 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.311+0000 7f485c811700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 0x7f4858113a90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4850001d00 tx=0x7f4850009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.312+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4850004020 con 0x7f4858102f50 2024-02-10T06:39:46.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.312+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485810dc30 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.312+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f485000f040 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.312+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4850009310 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.312+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485810e170 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.313+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4858105420 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.313+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4850009580 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.313+0000 7f48557fa700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48440692a0 0x7f484406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.314+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f48500868a0 con 0x7f4858102f50 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.314+0000 7f4857fff700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48440692a0 0x7f484406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.314+0000 7f4857fff700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48440692a0 0x7f484406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f485810f2c0 tx=0x7f4848009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:46.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.318+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f485004cbd0 con 0x7f4858102f50 2024-02-10T06:39:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.555+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f485804bed0 con 0x7f4858102f50 2024-02-10T06:39:46.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.559+0000 7f48557fa700 1 -- 172.21.15.73:0/946590496 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167171 (secure 0 0 0) 0x7f485004ad20 con 0x7f4858102f50 2024-02-10T06:39:46.562 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48440692a0 msgr2=0x7f484406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48440692a0 0x7f484406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f485810f2c0 tx=0x7f4848009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 msgr2=0x7f4858113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 0x7f4858113a90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4850001d00 tx=0x7f4850009f30 comp rx=0 tx=0).stop 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 shutdown_connections 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48440692a0 0x7f484406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:46.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4858102f50 0x7f4858113a90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:46.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 --2- 172.21.15.73:0/946590496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48581038f0 0x7f4858113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:46.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 >> 172.21.15.73:0/946590496 conn(0x7f48580f8ae0 msgr2=0x7f48580fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:46.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 shutdown_connections 2024-02-10T06:39:46.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:46.561+0000 7f485ea75700 1 -- 172.21.15.73:0/946590496 wait complete. 2024-02-10T06:39:47.197 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:46 smithi073 ceph-mon[29063]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:47.198 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:46 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/946590496' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:39:47.222 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":19,"active_gid":14212,"active_name":"smithi073.vjivfn","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":1890404867},{"type":"v1","addr":"172.21.15.73:6801","nonce":1890404867}]},"active_addr":"172.21.15.73:6801/1890404867","active_change":"2024-02-10T06:35:57.420823+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi181.lmidhc","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.73:8443/","prometheus":"http://172.21.15.73:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":4218001091}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":3977363692}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":1932431895}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":1987992948}]}]} 2024-02-10T06:39:47.226 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T06:39:47.226 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T06:39:47.226 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:39:47.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:46 smithi181 ceph-mon[36721]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:47.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:46 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/946590496' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:39:47.484 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:49.139 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:48 smithi073 ceph-mon[29063]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.136+0000 7fcd877f4700 1 -- 172.21.15.73:0/3089134706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 msgr2=0x7fcd800fe390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.136+0000 7fcd877f4700 1 --2- 172.21.15.73:0/3089134706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd800fe390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c004940 tx=0x7fcd7c009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- 172.21.15.73:0/3089134706 shutdown_connections 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 --2- 172.21.15.73:0/3089134706 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd800fe960 0x7fcd80106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 --2- 172.21.15.73:0/3089134706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd800fe390 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- 172.21.15.73:0/3089134706 >> 172.21.15.73:0/3089134706 conn(0x7fcd800f7a10 msgr2=0x7fcd800f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- 172.21.15.73:0/3089134706 shutdown_connections 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- 172.21.15.73:0/3089134706 wait complete. 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 Processor -- start 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- start start 2024-02-10T06:39:49.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd80113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd800fe960 0x7fcd80113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd801145a0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.137+0000 7fcd877f4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8010d8d0 con 0x7fcd800fe960 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd80113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd80113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59178/0 (socket says 172.21.15.73:59178) 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 -- 172.21.15.73:0/1163403536 learned_addr learned my addr 172.21.15.73:0/1163403536 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd84d8f700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd800fe960 0x7fcd80113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 -- 172.21.15.73:0/1163403536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd800fe960 msgr2=0x7fcd80113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd800fe960 0x7fcd80113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 -- 172.21.15.73:0/1163403536 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c005040 con 0x7fcd800fdfc0 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd85590700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd80113a10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c000c00 tx=0x7fcd7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:49.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd7c00e040 con 0x7fcd800fdfc0 2024-02-10T06:39:49.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd7c009af0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8010dbb0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd7c01c630 con 0x7fcd800fdfc0 2024-02-10T06:39:49.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.138+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8010e0f0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.140+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd801045b0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.140+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcd7c00bbf0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.140+0000 7fcd75ffb700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcd6c0692a0 0x7fcd6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:49.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.140+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcd7c0880a0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.140+0000 7fcd84d8f700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcd6c0692a0 0x7fcd6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:49.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.141+0000 7fcd84d8f700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcd6c0692a0 0x7fcd6c06b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd8010f240 tx=0x7fcd700043c0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:49.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.145+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcd7c058240 con 0x7fcd800fdfc0 2024-02-10T06:39:49.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.327+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd8004bed0 con 0x7fcd800fdfc0 2024-02-10T06:39:49.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.327+0000 7fcd75ffb700 1 -- 172.21.15.73:0/1163403536 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11070 (secure 0 0 0) 0x7fcd7c06ae60 con 0x7fcd800fdfc0 2024-02-10T06:39:49.329 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:39:49.329 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":41,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:33:49.428986+0000","modified":"2024-02-10T06:39:29.531882+0000","last_up_change":"2024-02-10T06:39:28.526237+0000","last_in_change":"2024-02-10T06:39:10.116034+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:38:17.748559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8e333d6f-68ed-41e7-a417-bed0ad6e2ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6803","nonce":1003470321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6805","nonce":1003470321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6809","nonce":1003470321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6807","nonce":1003470321}]},"public_addr":"172.21.15.73:6803/1003470321","cluster_addr":"172.21.15.73:6805/1003470321","heartbeat_back_addr":"172.21.15.73:6809/1003470321","heartbeat_front_addr":"172.21.15.73:6807/1003470321","state":["exists","up"]},{"osd":1,"uuid":"dedbb82b-7b4d-47eb-8621-2a35fa20a05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6811","nonce":2186552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6813","nonce":2186552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6817","nonce":2186552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6815","nonce":2186552820}]},"public_addr":"172.21.15.73:6811/2186552820","cluster_addr":"172.21.15.73:6813/2186552820","heartbeat_back_addr":"172.21.15.73:6817/2186552820","heartbeat_front_addr":"172.21.15.73:6815/2186552820","state":["exists","up"]},{"osd":2,"uuid":"6e7d61ae-a65b-43db-ac2d-aec724834f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6819","nonce":4245951128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6821","nonce":4245951128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6825","nonce":4245951128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6823","nonce":4245951128}]},"public_addr":"172.21.15.73:6819/4245951128","cluster_addr":"172.21.15.73:6821/4245951128","heartbeat_back_addr":"172.21.15.73:6825/4245951128","heartbeat_front_addr":"172.21.15.73:6823/4245951128","state":["exists","up"]},{"osd":3,"uuid":"2f17b271-99d5-4b38-89fb-175414c59351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6801","nonce":156965919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6803","nonce":156965919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6807","nonce":156965919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6805","nonce":156965919}]},"public_addr":"172.21.15.181:6801/156965919","cluster_addr":"172.21.15.181:6803/156965919","heartbeat_back_addr":"172.21.15.181:6807/156965919","heartbeat_front_addr":"172.21.15.181:6805/156965919","state":["exists","up"]},{"osd":4,"uuid":"28fd87c6-5f23-44af-87a6-d3cf81095f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6809","nonce":849727482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6811","nonce":849727482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6815","nonce":849727482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6813","nonce":849727482}]},"public_addr":"172.21.15.181:6809/849727482","cluster_addr":"172.21.15.181:6811/849727482","heartbeat_back_addr":"172.21.15.181:6815/849727482","heartbeat_front_addr":"172.21.15.181:6813/849727482","state":["exists","up"]},{"osd":5,"uuid":"9bd38e72-1e2b-41a3-974c-c5d431919187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6817","nonce":1157810956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6819","nonce":1157810956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6823","nonce":1157810956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6821","nonce":1157810956}]},"public_addr":"172.21.15.181:6817/1157810956","cluster_addr":"172.21.15.181:6819/1157810956","heartbeat_back_addr":"172.21.15.181:6823/1157810956","heartbeat_front_addr":"172.21.15.181:6821/1157810956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:14.313049+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:44.358563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:14.291417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:37.782333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:02.393199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:26.947433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:0/3920786161":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/4190786023":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6800/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/1749918597":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6801/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6801/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/3119085620":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/3298620834":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/1695656680":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/2932627761":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6800/194898351":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6800/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/1643478797":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/810500280":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6801/194898351":"2024-02-11T06:34:17.719875+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:39:49.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcd6c0692a0 msgr2=0x7fcd6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcd6c0692a0 0x7fcd6c06b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd8010f240 tx=0x7fcd700043c0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 msgr2=0x7fcd80113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd80113a10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c000c00 tx=0x7fcd7c009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 shutdown_connections 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcd6c0692a0 0x7fcd6c06b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd800fdfc0 0x7fcd80113a10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 --2- 172.21.15.73:0/1163403536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd800fe960 0x7fcd80113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.329+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 >> 172.21.15.73:0/1163403536 conn(0x7fcd800f7a10 msgr2=0x7fcd80101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.330+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 shutdown_connections 2024-02-10T06:39:49.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:49.330+0000 7fcd877f4700 1 -- 172.21.15.73:0/1163403536 wait complete. 2024-02-10T06:39:49.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:48 smithi181 ceph-mon[36721]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:50.069 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T06:39:50.069 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:39:50.325 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:50.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:49 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1163403536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:39:50.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:49 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1163403536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:39:51.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:50 smithi073 ceph-mon[29063]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:51.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:50 smithi181 ceph-mon[36721]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:52.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 -- 172.21.15.73:0/3791213350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 msgr2=0x7fd26c0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 --2- 172.21.15.73:0/3791213350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c0fbe70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd258004940 tx=0x7fd258009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 -- 172.21.15.73:0/3791213350 shutdown_connections 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 --2- 172.21.15.73:0/3791213350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c0fbe70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 --2- 172.21.15.73:0/3791213350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26c101e60 0x7fd26c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 -- 172.21.15.73:0/3791213350 >> 172.21.15.73:0/3791213350 conn(0x7fd26c0f7a10 msgr2=0x7fd26c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 -- 172.21.15.73:0/3791213350 shutdown_connections 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 -- 172.21.15.73:0/3791213350 wait complete. 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.101+0000 7fd2731db700 1 Processor -- start 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd2731db700 1 -- start start 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd2731db700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26c101e60 0x7fd26c113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:52.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd2731db700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd2731db700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c1146e0 con 0x7fd26c102800 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd2731db700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c10d980 con 0x7fd26c101e60 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59194/0 (socket says 172.21.15.73:59194) 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 -- 172.21.15.73:0/699329405 learned_addr learned my addr 172.21.15.73:0/699329405 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd270f77700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26c101e60 0x7fd26c113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 -- 172.21.15.73:0/699329405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26c101e60 msgr2=0x7fd26c113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26c101e60 0x7fd26c113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 -- 172.21.15.73:0/699329405 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258005040 con 0x7fd26c102800 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.102+0000 7fd26bfff700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c114000 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2580090d0 tx=0x7fd258009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.103+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd25800e040 con 0x7fd26c102800 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.103+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c10dc00 con 0x7fd26c102800 2024-02-10T06:39:52.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.103+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd25800bbf0 con 0x7fd26c102800 2024-02-10T06:39:52.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.103+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd25801c630 con 0x7fd26c102800 2024-02-10T06:39:52.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.103+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c10e140 con 0x7fd26c102800 2024-02-10T06:39:52.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.103+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c0fb610 con 0x7fd26c102800 2024-02-10T06:39:52.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.104+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd25800bd60 con 0x7fd26c102800 2024-02-10T06:39:52.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.104+0000 7fd2697fa700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd25c0692a0 0x7fd25c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:52.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.104+0000 7fd270f77700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd25c0692a0 0x7fd25c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:52.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.104+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd2580881b0 con 0x7fd26c102800 2024-02-10T06:39:52.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.105+0000 7fd270f77700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd25c0692a0 0x7fd25c06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd260006fd0 tx=0x7fd260008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:52.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.109+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd2580582b0 con 0x7fd26c102800 2024-02-10T06:39:52.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.291+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd26c04bed0 con 0x7fd26c102800 2024-02-10T06:39:52.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.292+0000 7fd2697fa700 1 -- 172.21.15.73:0/699329405 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11070 (secure 0 0 0) 0x7fd258013e60 con 0x7fd26c102800 2024-02-10T06:39:52.294 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:39:52.294 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":41,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:33:49.428986+0000","modified":"2024-02-10T06:39:29.531882+0000","last_up_change":"2024-02-10T06:39:28.526237+0000","last_in_change":"2024-02-10T06:39:10.116034+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:38:17.748559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8e333d6f-68ed-41e7-a417-bed0ad6e2ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6803","nonce":1003470321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6805","nonce":1003470321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6809","nonce":1003470321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6807","nonce":1003470321}]},"public_addr":"172.21.15.73:6803/1003470321","cluster_addr":"172.21.15.73:6805/1003470321","heartbeat_back_addr":"172.21.15.73:6809/1003470321","heartbeat_front_addr":"172.21.15.73:6807/1003470321","state":["exists","up"]},{"osd":1,"uuid":"dedbb82b-7b4d-47eb-8621-2a35fa20a05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6811","nonce":2186552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6813","nonce":2186552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6817","nonce":2186552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6815","nonce":2186552820}]},"public_addr":"172.21.15.73:6811/2186552820","cluster_addr":"172.21.15.73:6813/2186552820","heartbeat_back_addr":"172.21.15.73:6817/2186552820","heartbeat_front_addr":"172.21.15.73:6815/2186552820","state":["exists","up"]},{"osd":2,"uuid":"6e7d61ae-a65b-43db-ac2d-aec724834f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6819","nonce":4245951128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6821","nonce":4245951128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6825","nonce":4245951128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6823","nonce":4245951128}]},"public_addr":"172.21.15.73:6819/4245951128","cluster_addr":"172.21.15.73:6821/4245951128","heartbeat_back_addr":"172.21.15.73:6825/4245951128","heartbeat_front_addr":"172.21.15.73:6823/4245951128","state":["exists","up"]},{"osd":3,"uuid":"2f17b271-99d5-4b38-89fb-175414c59351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6801","nonce":156965919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6803","nonce":156965919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6807","nonce":156965919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6805","nonce":156965919}]},"public_addr":"172.21.15.181:6801/156965919","cluster_addr":"172.21.15.181:6803/156965919","heartbeat_back_addr":"172.21.15.181:6807/156965919","heartbeat_front_addr":"172.21.15.181:6805/156965919","state":["exists","up"]},{"osd":4,"uuid":"28fd87c6-5f23-44af-87a6-d3cf81095f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6809","nonce":849727482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6811","nonce":849727482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6815","nonce":849727482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6813","nonce":849727482}]},"public_addr":"172.21.15.181:6809/849727482","cluster_addr":"172.21.15.181:6811/849727482","heartbeat_back_addr":"172.21.15.181:6815/849727482","heartbeat_front_addr":"172.21.15.181:6813/849727482","state":["exists","up"]},{"osd":5,"uuid":"9bd38e72-1e2b-41a3-974c-c5d431919187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6817","nonce":1157810956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6819","nonce":1157810956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6823","nonce":1157810956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6821","nonce":1157810956}]},"public_addr":"172.21.15.181:6817/1157810956","cluster_addr":"172.21.15.181:6819/1157810956","heartbeat_back_addr":"172.21.15.181:6823/1157810956","heartbeat_front_addr":"172.21.15.181:6821/1157810956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:14.313049+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:44.358563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:14.291417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:37.782333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:02.393199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:26.947433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:0/3920786161":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/4190786023":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6800/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/1749918597":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6801/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6801/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/3119085620":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/3298620834":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/1695656680":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/2932627761":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6800/194898351":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6800/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/1643478797":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/810500280":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6801/194898351":"2024-02-11T06:34:17.719875+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:39:52.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd25c0692a0 msgr2=0x7fd25c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd25c0692a0 0x7fd25c06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd260006fd0 tx=0x7fd260008040 comp rx=0 tx=0).stop 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 msgr2=0x7fd26c114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c114000 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2580090d0 tx=0x7fd258009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 shutdown_connections 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd25c0692a0 0x7fd25c06b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26c101e60 0x7fd26c113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 --2- 172.21.15.73:0/699329405 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd26c102800 0x7fd26c114000 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 >> 172.21.15.73:0/699329405 conn(0x7fd26c0f7a10 msgr2=0x7fd26c065fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 shutdown_connections 2024-02-10T06:39:52.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:52.294+0000 7fd2731db700 1 -- 172.21.15.73:0/699329405 wait complete. 2024-02-10T06:39:53.054 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T06:39:53.054 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T06:39:53.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T06:39:53.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T06:39:53.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T06:39:53.056 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T06:39:53.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:52 smithi181 ceph-mon[36721]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:53.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:52 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/699329405' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:39:53.358 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:53.367 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:53.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:52 smithi073 ceph-mon[29063]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:53.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:52 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/699329405' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:39:55.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:54 smithi073 ceph-mon[29063]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:55.312 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:55.315 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:39:55.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:54 smithi181 ceph-mon[36721]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:56.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 -- 172.21.15.73:0/485889491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 msgr2=0x7f37800fb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:56.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 --2- 172.21.15.73:0/485889491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f37800fb510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3770004940 tx=0x7f3770009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:56.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 -- 172.21.15.73:0/485889491 shutdown_connections 2024-02-10T06:39:56.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 --2- 172.21.15.73:0/485889491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f37800fb510 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 --2- 172.21.15.73:0/485889491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3780102f70 0x7f3780103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 -- 172.21.15.73:0/485889491 >> 172.21.15.73:0/485889491 conn(0x7f3780071500 msgr2=0x7f3780071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 -- 172.21.15.73:0/485889491 shutdown_connections 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.273+0000 7f37886bd700 1 -- 172.21.15.73:0/485889491 wait complete. 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f37886bd700 1 Processor -- start 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f37886bd700 1 -- start start 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f37886bd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3780102f70 0x7f3780113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f37886bd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f3780113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f37886bd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3780114590 con 0x7f3780103910 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f37886bd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378010d8c0 con 0x7f3780102f70 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f3780113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f3780113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59220/0 (socket says 172.21.15.73:59220) 2024-02-10T06:39:56.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 -- 172.21.15.73:0/4004206095 learned_addr learned my addr 172.21.15.73:0/4004206095 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3786459700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3780102f70 0x7f3780113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 -- 172.21.15.73:0/4004206095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3780102f70 msgr2=0x7f3780113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3780102f70 0x7f3780113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 -- 172.21.15.73:0/4004206095 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770005040 con 0x7f3780103910 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3785c58700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f3780113f40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f37700090d0 tx=0x7f3770009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.274+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f377000e040 con 0x7f3780103910 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.275+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f377000bbf0 con 0x7f3780103910 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.275+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378010db40 con 0x7f3780103910 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.275+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f377001c630 con 0x7f3780103910 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.275+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378010e080 con 0x7f3780103910 2024-02-10T06:39:56.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.275+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f378010e6b0 con 0x7f3780103910 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f377000bd60 con 0x7f3780103910 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3776ffd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f376c0692a0 0x7f376c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3770088080 con 0x7f3780103910 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3786459700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f376c0692a0 0x7f376c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3776ffd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] conn(0x7f376c06ef70 0x7f376c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 --> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f376c071a30 con 0x7f376c06ef70 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.276+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f377008a6f0 con 0x7f3780103910 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.277+0000 7f3786c5a700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] conn(0x7f376c06ef70 0x7f376c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.277+0000 7f3786459700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f376c0692a0 0x7f376c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f377c009fd0 tx=0x7f377c009e70 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.277+0000 7f3786c5a700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] conn(0x7f376c06ef70 0x7f376c071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:56.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.278+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== osd.1 v2:172.21.15.73:6810/2186552820 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f376c071a30 con 0x7f376c06ef70 2024-02-10T06:39:56.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.292+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 --> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f378005e690 con 0x7f376c06ef70 2024-02-10T06:39:56.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.292+0000 7f3776ffd700 1 -- 172.21.15.73:0/4004206095 <== osd.1 v2:172.21.15.73:6810/2186552820 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f378005e690 con 0x7f376c06ef70 2024-02-10T06:39:56.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] conn(0x7f376c06ef70 msgr2=0x7f376c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:56.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] conn(0x7f376c06ef70 0x7f376c071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f376c0692a0 msgr2=0x7f376c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:56.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f376c0692a0 0x7f376c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f377c009fd0 tx=0x7f377c009e70 comp rx=0 tx=0).stop 2024-02-10T06:39:56.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 msgr2=0x7f3780113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f3780113f40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f37700090d0 tx=0x7f3770009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 shutdown_connections 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f376c0692a0 0x7f376c06b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:6810/2186552820,v1:172.21.15.73:6811/2186552820] conn(0x7f376c06ef70 0x7f376c071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3780102f70 0x7f3780113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 --2- 172.21.15.73:0/4004206095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3780103910 0x7f3780113f40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 >> 172.21.15.73:0/4004206095 conn(0x7f3780071500 msgr2=0x7f37800fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 shutdown_connections 2024-02-10T06:39:56.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:56.293+0000 7f37886bd700 1 -- 172.21.15.73:0/4004206095 wait complete. 2024-02-10T06:39:57.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:56 smithi073 ceph-mon[29063]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:57.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:56 smithi181 ceph-mon[36721]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:58.011 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:58.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:39:58.423 INFO:teuthology.orchestra.run.smithi073.stdout:60129542173 2024-02-10T06:39:58.424 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 -- 172.21.15.73:0/1342205424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 msgr2=0x7fb7100fe390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 --2- 172.21.15.73:0/1342205424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb7100fe390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb700004940 tx=0x7fb700009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 -- 172.21.15.73:0/1342205424 shutdown_connections 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 --2- 172.21.15.73:0/1342205424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7100fe960 0x7fb710106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 --2- 172.21.15.73:0/1342205424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb7100fe390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 -- 172.21.15.73:0/1342205424 >> 172.21.15.73:0/1342205424 conn(0x7fb7100f7a10 msgr2=0x7fb7100f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:59.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 -- 172.21.15.73:0/1342205424 shutdown_connections 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.063+0000 7fb715bc9700 1 -- 172.21.15.73:0/1342205424 wait complete. 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb715bc9700 1 Processor -- start 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb715bc9700 1 -- start start 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb715bc9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb710113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb715bc9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7100fe960 0x7fb710113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb715bc9700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101145a0 con 0x7fb7100fdfc0 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb715bc9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71010d8d0 con 0x7fb7100fe960 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb70f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb710113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb70f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb710113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53838/0 (socket says 172.21.15.73:53838) 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.064+0000 7fb70f7fe700 1 -- 172.21.15.73:0/3127674392 learned_addr learned my addr 172.21.15.73:0/3127674392 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb70effd700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7100fe960 0x7fb710113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:59.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb70f7fe700 1 -- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7100fe960 msgr2=0x7fb710113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb70f7fe700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7100fe960 0x7fb710113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb70f7fe700 1 -- 172.21.15.73:0/3127674392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700005040 con 0x7fb7100fdfc0 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb70f7fe700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb710113a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb700004890 tx=0x7fb700009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb70000e040 con 0x7fb7100fdfc0 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71010dbb0 con 0x7fb7100fdfc0 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb700004290 con 0x7fb7100fdfc0 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb70001c630 con 0x7fb7100fdfc0 2024-02-10T06:39:59.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.065+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71010e0f0 con 0x7fb7100fdfc0 2024-02-10T06:39:59.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.066+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb71010e590 con 0x7fb7100fdfc0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb70000bbf0 con 0x7fb7100fdfc0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb6f7fff700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb6f80692a0 0x7fb6f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb700088d40 con 0x7fb7100fdfc0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb70effd700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb6f80692a0 0x7fb6f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb6f7fff700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] conn(0x7fb6f806ef70 0x7fb6f8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 --> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6f8071a30 con 0x7fb6f806ef70 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb700089120 con 0x7fb7100fdfc0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.067+0000 7fb70effd700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb6f80692a0 0x7fb6f806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb704007fd0 tx=0x7fb704009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.068+0000 7fb70ffff700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] conn(0x7fb6f806ef70 0x7fb6f8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:39:59.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.068+0000 7fb70ffff700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] conn(0x7fb6f806ef70 0x7fb6f8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:39:59.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.069+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== osd.5 v2:172.21.15.181:6816/1157810956 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb6f8071a30 con 0x7fb6f806ef70 2024-02-10T06:39:59.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 --> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb71010e750 con 0x7fb6f806ef70 2024-02-10T06:39:59.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb6f7fff700 1 -- 172.21.15.73:0/3127674392 <== osd.5 v2:172.21.15.181:6816/1157810956 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb71010e750 con 0x7fb6f806ef70 2024-02-10T06:39:59.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] conn(0x7fb6f806ef70 msgr2=0x7fb6f8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:59.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] conn(0x7fb6f806ef70 0x7fb6f8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb6f80692a0 msgr2=0x7fb6f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb6f80692a0 0x7fb6f806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb704007fd0 tx=0x7fb704009040 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 msgr2=0x7fb710113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb710113a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb700004890 tx=0x7fb700009e30 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 shutdown_connections 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.084+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb6f80692a0 0x7fb6f806b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.085+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:6816/1157810956,v1:172.21.15.181:6817/1157810956] conn(0x7fb6f806ef70 0x7fb6f8071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.085+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7100fdfc0 0x7fb710113a10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.085+0000 7fb715bc9700 1 --2- 172.21.15.73:0/3127674392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7100fe960 0x7fb710113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.085+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 >> 172.21.15.73:0/3127674392 conn(0x7fb7100f7a10 msgr2=0x7fb710101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.085+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 shutdown_connections 2024-02-10T06:39:59.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:39:59.085+0000 7fb715bc9700 1 -- 172.21.15.73:0/3127674392 wait complete. 2024-02-10T06:39:59.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:39:58 smithi073 ceph-mon[29063]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:39:59.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:39:58 smithi181 ceph-mon[36721]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:01.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:00 smithi073 ceph-mon[29063]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:01.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:00 smithi073 ceph-mon[29063]: overall HEALTH_OK 2024-02-10T06:40:01.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:00 smithi181 ceph-mon[36721]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:01.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:00 smithi181 ceph-mon[36721]: overall HEALTH_OK 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 -- 172.21.15.73:0/2014868187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 msgr2=0x7f71f80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/2014868187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f80fe3b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f71e8004970 tx=0x7f71e8009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 -- 172.21.15.73:0/2014868187 shutdown_connections 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/2014868187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71f80fe8f0 0x7f71f8107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/2014868187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f80fe3b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 -- 172.21.15.73:0/2014868187 >> 172.21.15.73:0/2014868187 conn(0x7f71f8071d90 msgr2=0x7f71f8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:01.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 -- 172.21.15.73:0/2014868187 shutdown_connections 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.863+0000 7f71fd7e3700 1 -- 172.21.15.73:0/2014868187 wait complete. 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71fd7e3700 1 Processor -- start 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71fd7e3700 1 -- start start 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71fd7e3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f8113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71fd7e3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71f80fe8f0 0x7f71f8114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71fd7e3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f81146f0 con 0x7f71f80fdfe0 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71fd7e3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f810d990 con 0x7f71f80fe8f0 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71f6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f8113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71f6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f8113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53864/0 (socket says 172.21.15.73:53864) 2024-02-10T06:40:01.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.864+0000 7f71f6ffd700 1 -- 172.21.15.73:0/230740287 learned_addr learned my addr 172.21.15.73:0/230740287 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71f6ffd700 1 -- 172.21.15.73:0/230740287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71f80fe8f0 msgr2=0x7f71f8114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71f6ffd700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71f80fe8f0 0x7f71f8114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71f6ffd700 1 -- 172.21.15.73:0/230740287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8005040 con 0x7f71f80fdfe0 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71f6ffd700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f8113ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f71e8000c00 tx=0x7f71e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71e800e040 con 0x7f71f80fdfe0 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f810dc10 con 0x7f71f80fdfe0 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71e8004290 con 0x7f71f80fdfe0 2024-02-10T06:40:01.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71e801c630 con 0x7f71f80fdfe0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.865+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f810e150 con 0x7f71f80fdfe0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.866+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71f810e5f0 con 0x7f71f80fdfe0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.866+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f71e8013980 con 0x7f71f80fdfe0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71df7fe700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71e00692a0 0x7f71e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f71e8088140 con 0x7f71f80fdfe0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71f67fc700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71e00692a0 0x7f71e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71df7fe700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] conn(0x7f71e006ef70 0x7f71e0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 --> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71e0071a30 con 0x7f71e006ef70 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71e808a730 con 0x7f71f80fdfe0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71f77fe700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] conn(0x7f71e006ef70 0x7f71e0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71f67fc700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71e00692a0 0x7f71e006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f71ec006fd0 tx=0x7f71ec006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:01.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.867+0000 7f71f77fe700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] conn(0x7f71e006ef70 0x7f71e0071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:01.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.868+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== osd.0 v2:172.21.15.73:6802/1003470321 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f71e0071a30 con 0x7f71e006ef70 2024-02-10T06:40:01.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 --> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71f805e690 con 0x7f71e006ef70 2024-02-10T06:40:01.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71df7fe700 1 -- 172.21.15.73:0/230740287 <== osd.0 v2:172.21.15.73:6802/1003470321 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f71f805e690 con 0x7f71e006ef70 2024-02-10T06:40:01.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] conn(0x7f71e006ef70 msgr2=0x7f71e0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] conn(0x7f71e006ef70 0x7f71e0071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71e00692a0 msgr2=0x7f71e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71e00692a0 0x7f71e006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f71ec006fd0 tx=0x7f71ec006db0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 msgr2=0x7f71f8113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f8113ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f71e8000c00 tx=0x7f71e8009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 shutdown_connections 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6802/1003470321,v1:172.21.15.73:6803/1003470321] conn(0x7f71e006ef70 0x7f71e0071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71e00692a0 0x7f71e006b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71f80fdfe0 0x7f71f8113ad0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 --2- 172.21.15.73:0/230740287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71f80fe8f0 0x7f71f8114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 >> 172.21.15.73:0/230740287 conn(0x7f71f8071d90 msgr2=0x7f71f8101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.883+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 shutdown_connections 2024-02-10T06:40:01.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:01.884+0000 7f71fd7e3700 1 -- 172.21.15.73:0/230740287 wait complete. 2024-02-10T06:40:02.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 -- 172.21.15.73:0/3948298105 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c54102f40 msgr2=0x7f1c54103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:02.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3948298105 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c54102f40 0x7f1c54103310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c004940 tx=0x7f1c4c009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:02.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 -- 172.21.15.73:0/3948298105 shutdown_connections 2024-02-10T06:40:02.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3948298105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c541038e0 0x7f1c540fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3948298105 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c54102f40 0x7f1c54103310 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 -- 172.21.15.73:0/3948298105 >> 172.21.15.73:0/3948298105 conn(0x7f1c540f8af0 msgr2=0x7f1c540faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 -- 172.21.15.73:0/3948298105 shutdown_connections 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 -- 172.21.15.73:0/3948298105 wait complete. 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.089+0000 7f1c5a076700 1 Processor -- start 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 -- start start 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c54102f40 0x7f1c54113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 0x7f1c54113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54114680 con 0x7f1c541038e0 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5410d920 con 0x7f1c54102f40 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 0x7f1c54113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 0x7f1c54113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53884/0 (socket says 172.21.15.73:53884) 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 -- 172.21.15.73:0/3090026233 learned_addr learned my addr 172.21.15.73:0/3090026233 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:02.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c537fe700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c54102f40 0x7f1c54113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 -- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c54102f40 msgr2=0x7f1c54113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c54102f40 0x7f1c54113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 -- 172.21.15.73:0/3090026233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c005040 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c52ffd700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 0x7f1c54113fa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800b790 tx=0x7f1c48009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c4800f6f0 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5410dc00 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c4800fd20 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c480103a0 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.090+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5410e140 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.091+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c5410e720 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1c48011400 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c3bfff700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c3c0692a0 0x7f1c3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1c480876d0 con 0x7f1c541038e0 2024-02-10T06:40:02.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c537fe700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c3c0692a0 0x7f1c3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c3bfff700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] conn(0x7f1c3c06ef70 0x7f1c3c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 --> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c3c071a30 con 0x7f1c3c06ef70 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.092+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c48087ab0 con 0x7f1c541038e0 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.093+0000 7f1c537fe700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c3c0692a0 0x7f1c3c06b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c004f90 tx=0x7f1c4c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.093+0000 7f1c53fff700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] conn(0x7f1c3c06ef70 0x7f1c3c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.093+0000 7f1c53fff700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] conn(0x7f1c3c06ef70 0x7f1c3c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:02.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.094+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== osd.4 v2:172.21.15.181:6808/849727482 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1c3c071a30 con 0x7f1c3c06ef70 2024-02-10T06:40:02.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.108+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 --> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c540faec0 con 0x7f1c3c06ef70 2024-02-10T06:40:02.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c3bfff700 1 -- 172.21.15.73:0/3090026233 <== osd.4 v2:172.21.15.181:6808/849727482 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1c540faec0 con 0x7f1c3c06ef70 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] conn(0x7f1c3c06ef70 msgr2=0x7f1c3c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] conn(0x7f1c3c06ef70 0x7f1c3c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c3c0692a0 msgr2=0x7f1c3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c3c0692a0 0x7f1c3c06b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c004f90 tx=0x7f1c4c009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 msgr2=0x7f1c54113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.109+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 0x7f1c54113fa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800b790 tx=0x7f1c48009220 comp rx=0 tx=0).stop 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 shutdown_connections 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:6808/849727482,v1:172.21.15.181:6809/849727482] conn(0x7f1c3c06ef70 0x7f1c3c071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1c3c0692a0 0x7f1c3c06b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c54102f40 0x7f1c54113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 --2- 172.21.15.73:0/3090026233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1c541038e0 0x7f1c54113fa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:02.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 >> 172.21.15.73:0/3090026233 conn(0x7f1c540f8af0 msgr2=0x7f1c540fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:02.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 shutdown_connections 2024-02-10T06:40:02.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:02.110+0000 7f1c5a076700 1 -- 172.21.15.73:0/3090026233 wait complete. 2024-02-10T06:40:03.337 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:02 smithi181 ceph-mon[36721]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:03.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:02 smithi073 ceph-mon[29063]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:04.161 INFO:teuthology.orchestra.run.smithi073.stdout:171798691849 2024-02-10T06:40:04.162 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T06:40:04.719 INFO:teuthology.orchestra.run.smithi073.stdout:38654705700 2024-02-10T06:40:04.719 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T06:40:05.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:04 smithi073 ceph-mon[29063]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:05.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:04 smithi181 ceph-mon[36721]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:05.724 INFO:teuthology.orchestra.run.smithi073.stdout:150323855375 2024-02-10T06:40:05.724 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T06:40:05.914 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:05.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 -- 172.21.15.73:0/1442931992 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 msgr2=0x7fb3980fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 --2- 172.21.15.73:0/1442931992 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb3980fb9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb38c004940 tx=0x7fb38c009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 -- 172.21.15.73:0/1442931992 shutdown_connections 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 --2- 172.21.15.73:0/1442931992 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb3980fb9c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 --2- 172.21.15.73:0/1442931992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 0x7fb398103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 -- 172.21.15.73:0/1442931992 >> 172.21.15.73:0/1442931992 conn(0x7fb398071560 msgr2=0x7fb398071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 -- 172.21.15.73:0/1442931992 shutdown_connections 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.921+0000 7fb39e025700 1 -- 172.21.15.73:0/1442931992 wait complete. 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb39e025700 1 Processor -- start 2024-02-10T06:40:05.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb39e025700 1 -- start start 2024-02-10T06:40:05.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb39e025700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb398113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:05.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb39e025700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 0x7fb3981140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb39e025700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb398114720 con 0x7fb3980fb570 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb39e025700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39810da50 con 0x7fb398103140 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb3977fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb398113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb3977fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb398113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53898/0 (socket says 172.21.15.73:53898) 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.922+0000 7fb3977fe700 1 -- 172.21.15.73:0/1487641241 learned_addr learned my addr 172.21.15.73:0/1487641241 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb396ffd700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 0x7fb3981140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb3977fe700 1 -- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 msgr2=0x7fb3981140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb3977fe700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 0x7fb3981140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb3977fe700 1 -- 172.21.15.73:0/1487641241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c005040 con 0x7fb3980fb570 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb396ffd700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 0x7fb3981140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb3977fe700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb398113b90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb388002a50 tx=0x7fb3880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb388007cc0 con 0x7fb3980fb570 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb38800f040 con 0x7fb3980fb570 2024-02-10T06:40:05.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb388009640 con 0x7fb3980fb570 2024-02-10T06:40:05.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39810dd30 con 0x7fb3980fb570 2024-02-10T06:40:05.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.923+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39810e270 con 0x7fb3980fb570 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.924+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb39810e710 con 0x7fb3980fb570 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.925+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb3880097a0 con 0x7fb3980fb570 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.925+0000 7fb37ffff700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3800692a0 0x7fb38006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.925+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb388086c40 con 0x7fb3980fb570 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.925+0000 7fb396ffd700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3800692a0 0x7fb38006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.925+0000 7fb37ffff700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] conn(0x7fb38006ef70 0x7fb380071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.926+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 --> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb380071a30 con 0x7fb38006ef70 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.926+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb38808c2a0 con 0x7fb3980fb570 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.926+0000 7fb397fff700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] conn(0x7fb38006ef70 0x7fb380071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:05.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.926+0000 7fb396ffd700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3800692a0 0x7fb38006b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b3b0 tx=0x7fb38c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:05.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.926+0000 7fb397fff700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] conn(0x7fb38006ef70 0x7fb380071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:05.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.927+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== osd.3 v2:172.21.15.181:6800/156965919 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb380071a30 con 0x7fb38006ef70 2024-02-10T06:40:05.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.947+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 --> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb39810e880 con 0x7fb38006ef70 2024-02-10T06:40:05.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.947+0000 7fb37ffff700 1 -- 172.21.15.73:0/1487641241 <== osd.3 v2:172.21.15.181:6800/156965919 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb39810e880 con 0x7fb38006ef70 2024-02-10T06:40:05.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] conn(0x7fb38006ef70 msgr2=0x7fb380071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:05.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] conn(0x7fb38006ef70 0x7fb380071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3800692a0 msgr2=0x7fb38006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:05.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3800692a0 0x7fb38006b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b3b0 tx=0x7fb38c009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:05.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 msgr2=0x7fb398113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb398113b90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb388002a50 tx=0x7fb3880046b0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 shutdown_connections 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:6800/156965919,v1:172.21.15.181:6801/156965919] conn(0x7fb38006ef70 0x7fb380071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3800692a0 0x7fb38006b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3980fb570 0x7fb398113b90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 --2- 172.21.15.73:0/1487641241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb398103140 0x7fb3981140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 >> 172.21.15.73:0/1487641241 conn(0x7fb398071560 msgr2=0x7fb3980f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 shutdown_connections 2024-02-10T06:40:05.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:05.948+0000 7fb39e025700 1 -- 172.21.15.73:0/1487641241 wait complete. 2024-02-10T06:40:06.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 -- 172.21.15.73:0/379094463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fe960 msgr2=0x7f4c6c106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 --2- 172.21.15.73:0/379094463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fe960 0x7f4c6c106fa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004940 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 -- 172.21.15.73:0/379094463 shutdown_connections 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 --2- 172.21.15.73:0/379094463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fe960 0x7f4c6c106fa0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 --2- 172.21.15.73:0/379094463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c0fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 -- 172.21.15.73:0/379094463 >> 172.21.15.73:0/379094463 conn(0x7f4c6c0f7a10 msgr2=0x7f4c6c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.188+0000 7f4c70ade700 1 -- 172.21.15.73:0/379094463 shutdown_connections 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 -- 172.21.15.73:0/379094463 wait complete. 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 Processor -- start 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 -- start start 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe960 0x7f4c6c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:06.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c114700 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c70ade700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c10d9a0 con 0x7f4c6c0fe960 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c6a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c6a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53914/0 (socket says 172.21.15.73:53914) 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c6a59c700 1 -- 172.21.15.73:0/2842916531 learned_addr learned my addr 172.21.15.73:0/2842916531 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c6a59c700 1 -- 172.21.15.73:0/2842916531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe960 msgr2=0x7f4c6c114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c6a59c700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe960 0x7f4c6c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.189+0000 7f4c6a59c700 1 -- 172.21.15.73:0/2842916531 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60005040 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c6a59c700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c113ae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b790 tx=0x7f4c5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c00f770 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c5c00fda0 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c10dc80 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c00d310 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c10e1c0 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.190+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c6c10e6c0 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.191+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4c5c010420 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c5affd700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c540692a0 0x7f4c5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c69d9b700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c540692a0 0x7f4c5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4c5c087930 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c5affd700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] conn(0x7f4c5406ef70 0x7f4c54071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 --> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c54071a30 con 0x7f4c5406ef70 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c5c087d10 con 0x7f4c6c0fdfc0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c6ad9d700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] conn(0x7f4c5406ef70 0x7f4c54071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c69d9b700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c540692a0 0x7f4c5406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004890 tx=0x7f4c60009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.192+0000 7f4c6ad9d700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] conn(0x7f4c5406ef70 0x7f4c54071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:06.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.193+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== osd.2 v2:172.21.15.73:6818/4245951128 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4c54071a30 con 0x7f4c5406ef70 2024-02-10T06:40:06.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.207+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 --> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c6c05e610 con 0x7f4c5406ef70 2024-02-10T06:40:06.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c5affd700 1 -- 172.21.15.73:0/2842916531 <== osd.2 v2:172.21.15.73:6818/4245951128 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4c6c05e610 con 0x7f4c5406ef70 2024-02-10T06:40:06.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] conn(0x7f4c5406ef70 msgr2=0x7f4c54071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:06.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] conn(0x7f4c5406ef70 0x7f4c54071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c540692a0 msgr2=0x7f4c5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c540692a0 0x7f4c5406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004890 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 msgr2=0x7f4c6c113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c113ae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b790 tx=0x7f4c5c009250 comp rx=0 tx=0).stop 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 shutdown_connections 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c540692a0 0x7f4c5406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:6818/4245951128,v1:172.21.15.73:6819/4245951128] conn(0x7f4c5406ef70 0x7f4c54071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c6c0fdfc0 0x7f4c6c113ae0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 --2- 172.21.15.73:0/2842916531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe960 0x7f4c6c114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 >> 172.21.15.73:0/2842916531 conn(0x7f4c6c0f7a10 msgr2=0x7f4c6c101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 shutdown_connections 2024-02-10T06:40:06.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:06.208+0000 7f4c70ade700 1 -- 172.21.15.73:0/2842916531 wait complete. 2024-02-10T06:40:06.986 INFO:teuthology.orchestra.run.smithi073.stdout:81604378649 2024-02-10T06:40:06.987 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T06:40:07.335 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:06 smithi073 ceph-mon[29063]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:07.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:06 smithi181 ceph-mon[36721]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:07.758 INFO:teuthology.orchestra.run.smithi073.stdout:115964117011 2024-02-10T06:40:07.758 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T06:40:08.062 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:08.530 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:09.235 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:08 smithi073 ceph-mon[29063]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:09.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:08 smithi181 ceph-mon[36721]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:09.578 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:09.578 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:11.203 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:10 smithi073 ceph-mon[29063]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:11.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:10 smithi181 ceph-mon[36721]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:11.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 -- 172.21.15.73:0/2999387179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 msgr2=0x7f69e00fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:11.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 --2- 172.21.15.73:0/2999387179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e00fc1a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004940 tx=0x7f69d0009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:11.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 -- 172.21.15.73:0/2999387179 shutdown_connections 2024-02-10T06:40:11.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 --2- 172.21.15.73:0/2999387179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69e00fc6e0 0x7f69e0109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:11.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 --2- 172.21.15.73:0/2999387179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e00fc1a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 -- 172.21.15.73:0/2999387179 >> 172.21.15.73:0/2999387179 conn(0x7f69e0071d90 msgr2=0x7f69e0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 -- 172.21.15.73:0/2999387179 shutdown_connections 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 -- 172.21.15.73:0/2999387179 wait complete. 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 Processor -- start 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.919+0000 7f69e53b3700 1 -- start start 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69e53b3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e0115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69e53b3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69e00fc6e0 0x7f69e0116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69e53b3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0116840 con 0x7f69e00fbdd0 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69e53b3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e010fae0 con 0x7f69e00fc6e0 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e0115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e0115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47108/0 (socket says 172.21.15.73:47108) 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 -- 172.21.15.73:0/997051438 learned_addr learned my addr 172.21.15.73:0/997051438 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69d67fc700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69e00fc6e0 0x7f69e0116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:11.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 -- 172.21.15.73:0/997051438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69e00fc6e0 msgr2=0x7f69e0116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69e00fc6e0 0x7f69e0116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 -- 172.21.15.73:0/997051438 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0005040 con 0x7f69e00fbdd0 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69deffd700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e0115c20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009360 tx=0x7f69d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d000e040 con 0x7f69e00fbdd0 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69d000bb90 con 0x7f69e00fbdd0 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e010fd60 con 0x7f69e00fbdd0 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d001c630 con 0x7f69e00fbdd0 2024-02-10T06:40:11.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.920+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e01102a0 con 0x7f69e00fbdd0 2024-02-10T06:40:11.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.921+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e0106b70 con 0x7f69e00fbdd0 2024-02-10T06:40:11.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.922+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f69d000bd00 con 0x7f69e00fbdd0 2024-02-10T06:40:11.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.922+0000 7f69d7fff700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f69cc0692a0 0x7f69cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:11.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.922+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f69d00880a0 con 0x7f69e00fbdd0 2024-02-10T06:40:11.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.922+0000 7f69d67fc700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f69cc0692a0 0x7f69cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:11.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.923+0000 7f69d67fc700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f69cc0692a0 0x7f69cc06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69e0111450 tx=0x7f69c80091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:11.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:11.927+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f69d0058240 con 0x7f69e00fbdd0 2024-02-10T06:40:12.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.118+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f69e0111200 con 0x7f69e00fbdd0 2024-02-10T06:40:12.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.118+0000 7f69d7fff700 1 -- 172.21.15.73:0/997051438 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f69d0057dd0 con 0x7f69e00fbdd0 2024-02-10T06:40:12.119 INFO:teuthology.orchestra.run.smithi073.stdout:60129542176 2024-02-10T06:40:12.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f69cc0692a0 msgr2=0x7f69cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:12.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f69cc0692a0 0x7f69cc06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69e0111450 tx=0x7f69c80091a0 comp rx=0 tx=0).stop 2024-02-10T06:40:12.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 msgr2=0x7f69e0115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:12.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e0115c20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009360 tx=0x7f69d0009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:12.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 shutdown_connections 2024-02-10T06:40:12.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f69cc0692a0 0x7f69cc06b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:12.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69e00fbdd0 0x7f69e0115c20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:12.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 --2- 172.21.15.73:0/997051438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69e00fc6e0 0x7f69e0116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:12.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 >> 172.21.15.73:0/997051438 conn(0x7f69e0071d90 msgr2=0x7f69e00fa920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:12.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 shutdown_connections 2024-02-10T06:40:12.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:12.120+0000 7f69e53b3700 1 -- 172.21.15.73:0/997051438 wait complete. 2024-02-10T06:40:13.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:12 smithi073 ceph-mon[29063]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:13.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:12 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/997051438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:40:13.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:13.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-02-10T06:40:13.079 DEBUG:teuthology.parallel:result is None 2024-02-10T06:40:13.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:12 smithi181 ceph-mon[36721]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:13.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:12 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/997051438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:40:13.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:15.255 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:14 smithi073 ceph-mon[29063]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:15.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:14 smithi181 ceph-mon[36721]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:16.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 -- 172.21.15.73:0/2824980322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb380101e40 msgr2=0x7fb380102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 --2- 172.21.15.73:0/2824980322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb380101e40 0x7fb380102210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb370004940 tx=0x7fb370009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 -- 172.21.15.73:0/2824980322 shutdown_connections 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 --2- 172.21.15.73:0/2824980322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3801027e0 0x7fb3800fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 --2- 172.21.15.73:0/2824980322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb380101e40 0x7fb380102210 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 -- 172.21.15.73:0/2824980322 >> 172.21.15.73:0/2824980322 conn(0x7fb380071d90 msgr2=0x7fb380072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 -- 172.21.15.73:0/2824980322 shutdown_connections 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 -- 172.21.15.73:0/2824980322 wait complete. 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.395+0000 7fb386595700 1 Processor -- start 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb386595700 1 -- start start 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb386595700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb380101e40 0x7fb380112970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb386595700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 0x7fb380112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:16.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb386595700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb380113590 con 0x7fb3801027e0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb386595700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38010c830 con 0x7fb380101e40 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 0x7fb380112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 0x7fb380112eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47124/0 (socket says 172.21.15.73:47124) 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 -- 172.21.15.73:0/188506482 learned_addr learned my addr 172.21.15.73:0/188506482 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37ffff700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb380101e40 0x7fb380112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 -- 172.21.15.73:0/188506482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb380101e40 msgr2=0x7fb380112970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb380101e40 0x7fb380112970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 -- 172.21.15.73:0/188506482 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb370005040 con 0x7fb3801027e0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37f7fe700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 0x7fb380112eb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb37400b790 tx=0x7fb374009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37400f6f0 con 0x7fb3801027e0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb37400fd20 con 0x7fb3801027e0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.396+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3740103a0 con 0x7fb3801027e0 2024-02-10T06:40:16.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.397+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38010cb10 con 0x7fb3801027e0 2024-02-10T06:40:16.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.397+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38010d000 con 0x7fb3801027e0 2024-02-10T06:40:16.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.398+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3800738b0 con 0x7fb3801027e0 2024-02-10T06:40:16.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.398+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb374011400 con 0x7fb3801027e0 2024-02-10T06:40:16.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.398+0000 7fb37cff9700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3680692a0 0x7fb36806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:16.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.398+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb374086d10 con 0x7fb3801027e0 2024-02-10T06:40:16.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.398+0000 7fb37ffff700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3680692a0 0x7fb36806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:16.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.399+0000 7fb37ffff700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3680692a0 0x7fb36806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb370004f90 tx=0x7fb370009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:16.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.403+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb374056d90 con 0x7fb3801027e0 2024-02-10T06:40:16.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.593+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb38010df50 con 0x7fb3801027e0 2024-02-10T06:40:16.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.593+0000 7fb37cff9700 1 -- 172.21.15.73:0/188506482 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb374056920 con 0x7fb3801027e0 2024-02-10T06:40:16.594 INFO:teuthology.orchestra.run.smithi073.stdout:171798691853 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.595+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3680692a0 msgr2=0x7fb36806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.595+0000 7fb386595700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3680692a0 0x7fb36806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb370004f90 tx=0x7fb370009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.595+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 msgr2=0x7fb380112eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.595+0000 7fb386595700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 0x7fb380112eb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb37400b790 tx=0x7fb374009220 comp rx=0 tx=0).stop 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 shutdown_connections 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb3680692a0 0x7fb36806b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb380101e40 0x7fb380112970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 --2- 172.21.15.73:0/188506482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3801027e0 0x7fb380112eb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 >> 172.21.15.73:0/188506482 conn(0x7fb380071d90 msgr2=0x7fb3800fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:16.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 shutdown_connections 2024-02-10T06:40:16.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.596+0000 7fb386595700 1 -- 172.21.15.73:0/188506482 wait complete. 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- 172.21.15.73:0/2142147030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 msgr2=0x7fd740103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 --2- 172.21.15.73:0/2142147030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740103320 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd738009ce0 tx=0x7fd738004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- 172.21.15.73:0/2142147030 shutdown_connections 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 --2- 172.21.15.73:0/2142147030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7401038f0 0x7fd7400fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 --2- 172.21.15.73:0/2142147030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740103320 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- 172.21.15.73:0/2142147030 >> 172.21.15.73:0/2142147030 conn(0x7fd7400f8ae0 msgr2=0x7fd7400faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- 172.21.15.73:0/2142147030 shutdown_connections 2024-02-10T06:40:16.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- 172.21.15.73:0/2142147030 wait complete. 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 Processor -- start 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- start start 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740113c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7401038f0 0x7fd740114140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd740114820 con 0x7fd740102f50 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.966+0000 7fd744bd6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74010dae0 con 0x7fd7401038f0 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740113c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47146/0 (socket says 172.21.15.73:47146) 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 -- 172.21.15.73:0/1032134261 learned_addr learned my addr 172.21.15.73:0/1032134261 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73dd9b700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7401038f0 0x7fd740114140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 -- 172.21.15.73:0/1032134261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7401038f0 msgr2=0x7fd740114140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7401038f0 0x7fd740114140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:16.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 -- 172.21.15.73:0/1032134261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738009940 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd73e59c700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740113c00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd738004b70 tx=0x7fd738004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd73800e040 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd738004610 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd73801c810 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74010dd60 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.967+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74010e2a0 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.968+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7400fc6e0 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.969+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd738012c00 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.969+0000 7fd72effd700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd7280692b0 0x7fd72806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.969+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd738088ab0 con 0x7fd740102f50 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.969+0000 7fd73dd9b700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd7280692b0 0x7fd72806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.969+0000 7fd73dd9b700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd7280692b0 0x7fd72806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd734006fd0 tx=0x7fd734008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:16.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:16.973+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd7380550a0 con 0x7fd740102f50 2024-02-10T06:40:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:16 smithi073 ceph-mon[29063]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:16 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/188506482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:40:17.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.160+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd74010f280 con 0x7fd740102f50 2024-02-10T06:40:17.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.160+0000 7fd72effd700 1 -- 172.21.15.73:0/1032134261 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7380586c0 con 0x7fd740102f50 2024-02-10T06:40:17.161 INFO:teuthology.orchestra.run.smithi073.stdout:81604378651 2024-02-10T06:40:17.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd7280692b0 msgr2=0x7fd72806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:17.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd7280692b0 0x7fd72806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd734006fd0 tx=0x7fd734008040 comp rx=0 tx=0).stop 2024-02-10T06:40:17.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 msgr2=0x7fd740113c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:17.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740113c00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd738004b70 tx=0x7fd738004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:17.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 shutdown_connections 2024-02-10T06:40:17.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd7280692b0 0x7fd72806b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd740102f50 0x7fd740113c00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 --2- 172.21.15.73:0/1032134261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7401038f0 0x7fd740114140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 >> 172.21.15.73:0/1032134261 conn(0x7fd7400f8ae0 msgr2=0x7fd7400fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:17.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 shutdown_connections 2024-02-10T06:40:17.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.162+0000 7fd744bd6700 1 -- 172.21.15.73:0/1032134261 wait complete. 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 -- 172.21.15.73:0/2197377972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc1038e0 msgr2=0x7f49fc0fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 --2- 172.21.15.73:0/2197377972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc1038e0 0x7f49fc0fcf50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004940 tx=0x7f49f0009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 -- 172.21.15.73:0/2197377972 shutdown_connections 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 --2- 172.21.15.73:0/2197377972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc1038e0 0x7f49fc0fcf50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 --2- 172.21.15.73:0/2197377972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49fc102f40 0x7f49fc103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 -- 172.21.15.73:0/2197377972 >> 172.21.15.73:0/2197377972 conn(0x7f49fc0f8af0 msgr2=0x7f49fc0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 -- 172.21.15.73:0/2197377972 shutdown_connections 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.208+0000 7f4a00d22700 1 -- 172.21.15.73:0/2197377972 wait complete. 2024-02-10T06:40:17.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f4a00d22700 1 Processor -- start 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f4a00d22700 1 -- start start 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f4a00d22700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 0x7f49fc113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f4a00d22700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49fc1038e0 0x7f49fc114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f4a00d22700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc114730 con 0x7f49fc102f40 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f4a00d22700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc10d9d0 con 0x7f49fc1038e0 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49fa59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 0x7f49fc113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49fa59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 0x7f49fc113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47166/0 (socket says 172.21.15.73:47166) 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49fa59c700 1 -- 172.21.15.73:0/441673302 learned_addr learned my addr 172.21.15.73:0/441673302 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49f9d9b700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49fc1038e0 0x7f49fc114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49fa59c700 1 -- 172.21.15.73:0/441673302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49fc1038e0 msgr2=0x7f49fc114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49fa59c700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49fc1038e0 0x7f49fc114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.209+0000 7f49fa59c700 1 -- 172.21.15.73:0/441673302 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f0005040 con 0x7f49fc102f40 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.210+0000 7f49fa59c700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 0x7f49fc113b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00b790 tx=0x7f49ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:17.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.210+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49ec00f770 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.210+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49fc10dcb0 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.210+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49ec00fda0 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.210+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49ec00d310 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.210+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49fc10e1f0 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.211+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49fc100250 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.211+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f49ec010420 con 0x7f49fc102f40 2024-02-10T06:40:17.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.211+0000 7f49eaffd700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f49e40692a0 0x7f49e406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:17.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.212+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f49ec086e70 con 0x7f49fc102f40 2024-02-10T06:40:17.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.212+0000 7f49f9d9b700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f49e40692a0 0x7f49e406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:17.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.212+0000 7f49f9d9b700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f49e40692a0 0x7f49e406b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49fc10f340 tx=0x7f49f0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:17.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.216+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f49ec056f70 con 0x7f49fc102f40 2024-02-10T06:40:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:16 smithi181 ceph-mon[36721]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:16 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/188506482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:40:17.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.402+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f49fc04bed0 con 0x7f49fc102f40 2024-02-10T06:40:17.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.403+0000 7f49eaffd700 1 -- 172.21.15.73:0/441673302 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f49ec056b00 con 0x7f49fc102f40 2024-02-10T06:40:17.404 INFO:teuthology.orchestra.run.smithi073.stdout:115964117012 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f49e40692a0 msgr2=0x7f49e406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f49e40692a0 0x7f49e406b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49fc10f340 tx=0x7f49f0009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 msgr2=0x7f49fc113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 0x7f49fc113b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00b790 tx=0x7f49ec009250 comp rx=0 tx=0).stop 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 shutdown_connections 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f49e40692a0 0x7f49e406b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f49fc102f40 0x7f49fc113b10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 --2- 172.21.15.73:0/441673302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49fc1038e0 0x7f49fc114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:17.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 >> 172.21.15.73:0/441673302 conn(0x7f49fc0f8af0 msgr2=0x7f49fc0fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:17.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 shutdown_connections 2024-02-10T06:40:17.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:17.405+0000 7f4a00d22700 1 -- 172.21.15.73:0/441673302 wait complete. 2024-02-10T06:40:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:17 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1032134261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:40:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:17 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/441673302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:40:18.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:17 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1032134261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:40:18.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:17 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/441673302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:40:18.507 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691853 for osd.5 2024-02-10T06:40:18.507 DEBUG:teuthology.parallel:result is None 2024-02-10T06:40:18.993 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-02-10T06:40:18.994 DEBUG:teuthology.parallel:result is None 2024-02-10T06:40:19.334 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:18 smithi073 ceph-mon[29063]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:19.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:18 smithi181 ceph-mon[36721]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:19.362 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117011 got 115964117012 for osd.3 2024-02-10T06:40:19.362 DEBUG:teuthology.parallel:result is None 2024-02-10T06:40:20.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.055+0000 7fe6168fe700 1 -- 172.21.15.73:0/230554728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 msgr2=0x7fe610103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.055+0000 7fe6168fe700 1 --2- 172.21.15.73:0/230554728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610103320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe600004940 tx=0x7fe600009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:20.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.055+0000 7fe6168fe700 1 -- 172.21.15.73:0/230554728 shutdown_connections 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.055+0000 7fe6168fe700 1 --2- 172.21.15.73:0/230554728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6101038f0 0x7fe6100fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.055+0000 7fe6168fe700 1 --2- 172.21.15.73:0/230554728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610103320 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.055+0000 7fe6168fe700 1 -- 172.21.15.73:0/230554728 >> 172.21.15.73:0/230554728 conn(0x7fe6100f8ae0 msgr2=0x7fe6100faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 -- 172.21.15.73:0/230554728 shutdown_connections 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 -- 172.21.15.73:0/230554728 wait complete. 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 Processor -- start 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 -- start start 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6101038f0 0x7fe610114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610114700 con 0x7fe610102f50 2024-02-10T06:40:20.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe6168fe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61010d9a0 con 0x7fe6101038f0 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.056+0000 7fe60ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47944/0 (socket says 172.21.15.73:47944) 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60ffff700 1 -- 172.21.15.73:0/3722835454 learned_addr learned my addr 172.21.15.73:0/3722835454 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60f7fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6101038f0 0x7fe610114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60ffff700 1 -- 172.21.15.73:0/3722835454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6101038f0 msgr2=0x7fe610114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60ffff700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6101038f0 0x7fe610114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60ffff700 1 -- 172.21.15.73:0/3722835454 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe600005040 con 0x7fe610102f50 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60ffff700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610113ae0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe60000b3b0 tx=0x7fe600009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe60000e040 con 0x7fe610102f50 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61010dc20 con 0x7fe610102f50 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe60000bbf0 con 0x7fe610102f50 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe60001c630 con 0x7fe610102f50 2024-02-10T06:40:20.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.057+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61010e160 con 0x7fe610102f50 2024-02-10T06:40:20.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.058+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6100fc6e0 con 0x7fe610102f50 2024-02-10T06:40:20.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.059+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe60000bd60 con 0x7fe610102f50 2024-02-10T06:40:20.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.059+0000 7fe60cff9700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe5f80692a0 0x7fe5f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:20.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.059+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe600088250 con 0x7fe610102f50 2024-02-10T06:40:20.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.059+0000 7fe60f7fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe5f80692a0 0x7fe5f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:20.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.059+0000 7fe60f7fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe5f80692a0 0x7fe5f806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe604006fd0 tx=0x7fe604008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:20.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.063+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe600058350 con 0x7fe610102f50 2024-02-10T06:40:20.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.249+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe61010f0c0 con 0x7fe610102f50 2024-02-10T06:40:20.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.250+0000 7fe60cff9700 1 -- 172.21.15.73:0/3722835454 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe600057ee0 con 0x7fe610102f50 2024-02-10T06:40:20.251 INFO:teuthology.orchestra.run.smithi073.stdout:150323855378 2024-02-10T06:40:20.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe5f80692a0 msgr2=0x7fe5f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe5f80692a0 0x7fe5f806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe604006fd0 tx=0x7fe604008040 comp rx=0 tx=0).stop 2024-02-10T06:40:20.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 msgr2=0x7fe610113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610113ae0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe60000b3b0 tx=0x7fe600009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:20.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 shutdown_connections 2024-02-10T06:40:20.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe5f80692a0 0x7fe5f806b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe610102f50 0x7fe610113ae0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 --2- 172.21.15.73:0/3722835454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6101038f0 0x7fe610114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 >> 172.21.15.73:0/3722835454 conn(0x7fe6100f8ae0 msgr2=0x7fe6100fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:20.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 shutdown_connections 2024-02-10T06:40:20.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.252+0000 7fe6168fe700 1 -- 172.21.15.73:0/3722835454 wait complete. 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 -- 172.21.15.73:0/377619310 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 msgr2=0x7ff0540fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 --2- 172.21.15.73:0/377619310 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff0540fbea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff044009ce0 tx=0x7ff044004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 -- 172.21.15.73:0/377619310 shutdown_connections 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 --2- 172.21.15.73:0/377619310 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff0540fbea0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 --2- 172.21.15.73:0/377619310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 0x7ff054102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 -- 172.21.15.73:0/377619310 >> 172.21.15.73:0/377619310 conn(0x7ff054071d90 msgr2=0x7ff054072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:20.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 -- 172.21.15.73:0/377619310 shutdown_connections 2024-02-10T06:40:20.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.343+0000 7ff05af43700 1 -- 172.21.15.73:0/377619310 wait complete. 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff05af43700 1 Processor -- start 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff05af43700 1 -- start start 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff05af43700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 0x7ff0541119b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff05af43700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff054111ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff05af43700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff054112540 con 0x7ff0541027e0 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff05af43700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05410b870 con 0x7ff054101e40 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff053fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff054111ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff053fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff054111ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47966/0 (socket says 172.21.15.73:47966) 2024-02-10T06:40:20.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff053fff700 1 -- 172.21.15.73:0/2975523512 learned_addr learned my addr 172.21.15.73:0/2975523512 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff058cdf700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 0x7ff0541119b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff053fff700 1 -- 172.21.15.73:0/2975523512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 msgr2=0x7ff0541119b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff053fff700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 0x7ff0541119b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff053fff700 1 -- 172.21.15.73:0/2975523512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044009940 con 0x7ff0541027e0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.344+0000 7ff058cdf700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 0x7ff0541119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.345+0000 7ff053fff700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff054111ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff04400bea0 tx=0x7ff044004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.345+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04400e040 con 0x7ff0541027e0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.345+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05410baf0 con 0x7ff0541027e0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.345+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff044008ed0 con 0x7ff0541027e0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.345+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04401c8e0 con 0x7ff0541027e0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.345+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05410c030 con 0x7ff0541027e0 2024-02-10T06:40:20.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.346+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff054104310 con 0x7ff0541027e0 2024-02-10T06:40:20.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.346+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff044012410 con 0x7ff0541027e0 2024-02-10T06:40:20.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.347+0000 7ff0517fa700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff04006b4a0 0x7ff04006d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:20.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.347+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff044088b50 con 0x7ff0541027e0 2024-02-10T06:40:20.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.347+0000 7ff058cdf700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff04006b4a0 0x7ff04006d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:20.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.347+0000 7ff058cdf700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff04006b4a0 0x7ff04006d950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0041d0 tx=0x7ff04c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:20.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.351+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff044058bd0 con 0x7ff0541027e0 2024-02-10T06:40:20.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.535+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff0540738b0 con 0x7ff0541027e0 2024-02-10T06:40:20.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.535+0000 7ff0517fa700 1 -- 172.21.15.73:0/2975523512 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff044058760 con 0x7ff0541027e0 2024-02-10T06:40:20.536 INFO:teuthology.orchestra.run.smithi073.stdout:38654705704 2024-02-10T06:40:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.537+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff04006b4a0 msgr2=0x7ff04006d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.537+0000 7ff05af43700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff04006b4a0 0x7ff04006d950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0041d0 tx=0x7ff04c009040 comp rx=0 tx=0).stop 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.537+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 msgr2=0x7ff054111ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.537+0000 7ff05af43700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff054111ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff04400bea0 tx=0x7ff044004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 shutdown_connections 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7ff04006b4a0 0x7ff04006d950 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff054101e40 0x7ff0541119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 --2- 172.21.15.73:0/2975523512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff0541027e0 0x7ff054111ef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 >> 172.21.15.73:0/2975523512 conn(0x7ff054071d90 msgr2=0x7ff0540fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 shutdown_connections 2024-02-10T06:40:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:20.538+0000 7ff05af43700 1 -- 172.21.15.73:0/2975523512 wait complete. 2024-02-10T06:40:21.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:20 smithi073 ceph-mon[29063]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:21.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:20 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3722835454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:40:21.256 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:20 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2975523512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:40:21.281 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705704 for osd.0 2024-02-10T06:40:21.281 DEBUG:teuthology.parallel:result is None 2024-02-10T06:40:21.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:20 smithi181 ceph-mon[36721]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:21.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:20 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3722835454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:40:21.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:20 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2975523512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:40:21.871 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855378 for osd.4 2024-02-10T06:40:21.872 DEBUG:teuthology.parallel:result is None 2024-02-10T06:40:21.872 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T06:40:21.872 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:40:22.133 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:23.232 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:22 smithi073 ceph-mon[29063]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:23.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:22 smithi181 ceph-mon[36721]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 -- 172.21.15.73:0/713571981 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 msgr2=0x7f60f4103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/713571981 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f4103510 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f60e4004940 tx=0x7f60e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 -- 172.21.15.73:0/713571981 shutdown_connections 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/713571981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f40fb570 0x7f60f40fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/713571981 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f4103510 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 -- 172.21.15.73:0/713571981 >> 172.21.15.73:0/713571981 conn(0x7f60f4071560 msgr2=0x7f60f4071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 -- 172.21.15.73:0/713571981 shutdown_connections 2024-02-10T06:40:23.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.792+0000 7f60fbd4d700 1 -- 172.21.15.73:0/713571981 wait complete. 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60fbd4d700 1 Processor -- start 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60fbd4d700 1 -- start start 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60fbd4d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f40fb570 0x7f60f40fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60fbd4d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f40fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60fbd4d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f40fd290 con 0x7f60f4103140 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60fbd4d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f40fd400 con 0x7f60f40fb570 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f40fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f40fcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47984/0 (socket says 172.21.15.73:47984) 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 -- 172.21.15.73:0/2842443449 learned_addr learned my addr 172.21.15.73:0/2842443449 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f9ae9700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f40fb570 0x7f60f40fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 -- 172.21.15.73:0/2842443449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f40fb570 msgr2=0x7f60f40fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f40fb570 0x7f60f40fc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:23.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 -- 172.21.15.73:0/2842443449 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e4005040 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.793+0000 7f60f92e8700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f40fcc90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c830 tx=0x7f60f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.794+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60f000a760 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.794+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60f000ad90 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.794+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f4119d50 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.794+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60f00113a0 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.794+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f411a230 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.795+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f4100310 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.795+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f60f0012400 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.795+0000 7f60ea7fc700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60e00692a0 0x7f60e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.795+0000 7f60f9ae9700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60e00692a0 0x7f60e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.795+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f60f0086eb0 con 0x7f60f4103140 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.796+0000 7f60f9ae9700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60e00692a0 0x7f60e006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f60e4004f90 tx=0x7f60e4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:23.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.800+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f60f0053520 con 0x7f60f4103140 2024-02-10T06:40:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.978+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f60f404bf70 con 0x7f60e00692a0 2024-02-10T06:40:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.980+0000 7f60ea7fc700 1 -- 172.21.15.73:0/2842443449 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28078 (secure 0 0 0) 0x7f60e400b5c0 con 0x7f60e00692a0 2024-02-10T06:40:23.981 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60e00692a0 msgr2=0x7f60e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60e00692a0 0x7f60e006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f60e4004f90 tx=0x7f60e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 msgr2=0x7f60f40fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f40fcc90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c830 tx=0x7f60f0004470 comp rx=0 tx=0).stop 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 shutdown_connections 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60e00692a0 0x7f60e006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f40fb570 0x7f60f40fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 --2- 172.21.15.73:0/2842443449 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60f4103140 0x7f60f40fcc90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 >> 172.21.15.73:0/2842443449 conn(0x7f60f4071560 msgr2=0x7f60f40fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 shutdown_connections 2024-02-10T06:40:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:23.982+0000 7f60fbd4d700 1 -- 172.21.15.73:0/2842443449 wait complete. 2024-02-10T06:40:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-02-10T06:40:24.657 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-02-10T06:40:23.472630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002475"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-10T06:39:28.529041+0000","last_change":"2024-02-10T06:39:03.505582+0000","last_active":"2024-02-10T06:39:28.529041+0000","last_peered":"2024-02-10T06:39:28.529041+0000","last_clean":"2024-02-10T06:39:28.529041+0000","last_became_active":"2024-02-10T06:39:03.505203+0000","last_became_peered":"2024-02-10T06:39:03.505203+0000","last_unstale":"2024-02-10T06:39:28.529041+0000","last_undegraded":"2024-02-10T06:39:28.529041+0000","last_fullsized":"2024-02-10T06:39:28.529041+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T06:38:18.318172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T06:38:18.318172+0000","last_clean_scrub_stamp":"2024-02-10T06:38:18.318172+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:33:49.024789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.623},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.38400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40600000000000003}]},{"osd":1,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46800000000000003}]},{"osd":4,"last update":"Sat Feb 10 06:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.436},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.33900000000000002,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.442,"5min":0.45500000000000002,"15min":0.45500000000000002},"last":0.379},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38400000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.495,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.29599999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.26100000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.42099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.311,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.254}]},{"osd":2,"last update":"Sat Feb 10 06:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34599999999999997,"15min":0.34999999999999998},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.41099999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53500000000000003}]},{"osd":4,"last update":"Sat Feb 10 06:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.307,"15min":0.29999999999999999},"min":{"1min":0.22,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.25900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:39:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Feb 10 06:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.628}]},{"osd":4,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:40:24.658 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:40:24.913 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:25.252 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:24 smithi073 ceph-mon[29063]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:25.252 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:24 smithi073 ceph-mon[29063]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:40:25.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:24 smithi181 ceph-mon[36721]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:25.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:24 smithi181 ceph-mon[36721]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:40:26.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 -- 172.21.15.73:0/529274724 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 msgr2=0x7f867c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:26.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 --2- 172.21.15.73:0/529274724 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c102230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8674009ce0 tx=0x7f8674004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:26.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 -- 172.21.15.73:0/529274724 shutdown_connections 2024-02-10T06:40:26.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 --2- 172.21.15.73:0/529274724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f867c102800 0x7f867c0fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:26.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 --2- 172.21.15.73:0/529274724 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c102230 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 -- 172.21.15.73:0/529274724 >> 172.21.15.73:0/529274724 conn(0x7f867c0f7a10 msgr2=0x7f867c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 -- 172.21.15.73:0/529274724 shutdown_connections 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 -- 172.21.15.73:0/529274724 wait complete. 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 Processor -- start 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 -- start start 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.497+0000 7f8681545700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f8681545700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f867c102800 0x7f867c1140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f8681545700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c1147d0 con 0x7f867c101e60 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f8681545700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c10dae0 con 0x7f867c102800 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47996/0 (socket says 172.21.15.73:47996) 2024-02-10T06:40:26.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 -- 172.21.15.73:0/2342671282 learned_addr learned my addr 172.21.15.73:0/2342671282 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867a7fc700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f867c102800 0x7f867c1140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 -- 172.21.15.73:0/2342671282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f867c102800 msgr2=0x7f867c1140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f867c102800 0x7f867c1140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 -- 172.21.15.73:0/2342671282 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674009940 con 0x7f867c101e60 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f867affd700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c113bb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f867400bf60 tx=0x7f8674004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f867400e040 con 0x7f867c101e60 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86740128e0 con 0x7f867c101e60 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c10dd60 con 0x7f867c101e60 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f867401c870 con 0x7f867c101e60 2024-02-10T06:40:26.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.498+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c10e2a0 con 0x7f867c101e60 2024-02-10T06:40:26.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.499+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c0fb610 con 0x7f867c101e60 2024-02-10T06:40:26.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.500+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8674012a50 con 0x7f867c101e60 2024-02-10T06:40:26.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.500+0000 7f86637fe700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f86640692a0 0x7f866406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:26.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.500+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8674088bc0 con 0x7f867c101e60 2024-02-10T06:40:26.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.500+0000 7f867a7fc700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f86640692a0 0x7f866406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:26.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.500+0000 7f867a7fc700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f86640692a0 0x7f866406b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8670006fd0 tx=0x7f8670008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:26.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.504+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8674058c40 con 0x7f867c101e60 2024-02-10T06:40:26.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.686+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f867c04bf40 con 0x7f86640692a0 2024-02-10T06:40:26.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.687+0000 7f86637fe700 1 -- 172.21.15.73:0/2342671282 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28078 (secure 0 0 0) 0x7f867c04bf40 con 0x7f86640692a0 2024-02-10T06:40:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:40:26.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f86640692a0 msgr2=0x7f866406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:26.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f86640692a0 0x7f866406b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8670006fd0 tx=0x7f8670008040 comp rx=0 tx=0).stop 2024-02-10T06:40:26.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 msgr2=0x7f867c113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:26.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c113bb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f867400bf60 tx=0x7f8674004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 shutdown_connections 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f86640692a0 0x7f866406b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f867c101e60 0x7f867c113bb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 --2- 172.21.15.73:0/2342671282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f867c102800 0x7f867c1140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.689+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 >> 172.21.15.73:0/2342671282 conn(0x7f867c0f7a10 msgr2=0x7f867c065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.690+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 shutdown_connections 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:26.690+0000 7f8681545700 1 -- 172.21.15.73:0/2342671282 wait complete. 2024-02-10T06:40:26.691 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-02-10T06:40:27.297 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:26 smithi073 ceph-mon[29063]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:27.324 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-02-10T06:40:25.473054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-10T06:39:28.529041+0000","last_change":"2024-02-10T06:39:03.505582+0000","last_active":"2024-02-10T06:39:28.529041+0000","last_peered":"2024-02-10T06:39:28.529041+0000","last_clean":"2024-02-10T06:39:28.529041+0000","last_became_active":"2024-02-10T06:39:03.505203+0000","last_became_peered":"2024-02-10T06:39:03.505203+0000","last_unstale":"2024-02-10T06:39:28.529041+0000","last_undegraded":"2024-02-10T06:39:28.529041+0000","last_fullsized":"2024-02-10T06:39:28.529041+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T06:38:18.318172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T06:38:18.318172+0000","last_clean_scrub_stamp":"2024-02-10T06:38:18.318172+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:33:49.024789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]}]},{"osd":4,"up_from":35,"seq":150323855380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40600000000000003}]},{"osd":1,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46800000000000003}]},{"osd":4,"last update":"Sat Feb 10 06:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.436},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.33900000000000002,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.442,"5min":0.45500000000000002,"15min":0.45500000000000002},"last":0.379},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38400000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.495,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.29599999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.26100000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.42099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.311,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.254}]},{"osd":2,"last update":"Sat Feb 10 06:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34599999999999997,"15min":0.34999999999999998},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.41099999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53500000000000003}]},{"osd":4,"last update":"Sat Feb 10 06:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.307,"15min":0.29999999999999999},"min":{"1min":0.22,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.25900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:39:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Feb 10 06:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.628}]},{"osd":4,"last update":"Sat Feb 10 06:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:40:27.325 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T06:40:27.325 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T06:40:27.325 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T06:40:27.326 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T06:40:27.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:26 smithi181 ceph-mon[36721]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:27.580 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:28.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:27 smithi181 ceph-mon[36721]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:40:28.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:28.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:27 smithi073 ceph-mon[29063]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:40:28.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:29.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 -- 172.21.15.73:0/4186368563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 msgr2=0x7f2d400fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:29.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/4186368563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d400fbea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c004940 tx=0x7f2d2c009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:29.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 -- 172.21.15.73:0/4186368563 shutdown_connections 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/4186368563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d400fbea0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/4186368563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 0x7f2d40102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 -- 172.21.15.73:0/4186368563 >> 172.21.15.73:0/4186368563 conn(0x7f2d40071d90 msgr2=0x7f2d40072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 -- 172.21.15.73:0/4186368563 shutdown_connections 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.172+0000 7f2d46c0b700 1 -- 172.21.15.73:0/4186368563 wait complete. 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d46c0b700 1 Processor -- start 2024-02-10T06:40:29.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d46c0b700 1 -- start start 2024-02-10T06:40:29.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d46c0b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 0x7f2d40115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d46c0b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d40116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d46c0b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d401167a0 con 0x7f2d401027e0 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d46c0b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4010fb60 con 0x7f2d40101e40 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d3ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d40116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d3ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d40116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47104/0 (socket says 172.21.15.73:47104) 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d3ffff700 1 -- 172.21.15.73:0/1156835333 learned_addr learned my addr 172.21.15.73:0/1156835333 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d3ffff700 1 -- 172.21.15.73:0/1156835333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 msgr2=0x7f2d40115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d449a7700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 0x7f2d40115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.173+0000 7f2d3ffff700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 0x7f2d40115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d3ffff700 1 -- 172.21.15.73:0/1156835333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c005040 con 0x7f2d401027e0 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d449a7700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 0x7f2d40115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d3ffff700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d40116150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009150 tx=0x7f2d2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2c00e040 con 0x7f2d401027e0 2024-02-10T06:40:29.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4010fde0 con 0x7f2d401027e0 2024-02-10T06:40:29.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d2c00bbf0 con 0x7f2d401027e0 2024-02-10T06:40:29.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2c01c630 con 0x7f2d401027e0 2024-02-10T06:40:29.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:28 smithi073 ceph-mon[29063]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.174+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d40110320 con 0x7f2d401027e0 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.175+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d400738b0 con 0x7f2d401027e0 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.175+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2d2c00bd60 con 0x7f2d401027e0 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.176+0000 7f2d3d7fa700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f2d300692a0 0x7f2d3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.176+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2d2c088250 con 0x7f2d401027e0 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.176+0000 7f2d449a7700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f2d300692a0 0x7f2d3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.176+0000 7f2d449a7700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f2d300692a0 0x7f2d3006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d34007fd0 tx=0x7f2d34009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:29.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.180+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2d2c0548c0 con 0x7f2d401027e0 2024-02-10T06:40:29.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:28 smithi181 ceph-mon[36721]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:29.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.407+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2d40108f50 con 0x7f2d401027e0 2024-02-10T06:40:29.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.408+0000 7f2d3d7fa700 1 -- 172.21.15.73:0/1156835333 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2d2c057ee0 con 0x7f2d401027e0 2024-02-10T06:40:29.408 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:40:29.409 INFO:teuthology.orchestra.run.smithi073.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.409+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f2d300692a0 msgr2=0x7f2d3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.409+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f2d300692a0 0x7f2d3006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d34007fd0 tx=0x7f2d34009040 comp rx=0 tx=0).stop 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.409+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 msgr2=0x7f2d40116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.409+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d40116150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009150 tx=0x7f2d2c009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 shutdown_connections 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f2d300692a0 0x7f2d3006b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d40101e40 0x7f2d40115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 --2- 172.21.15.73:0/1156835333 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d401027e0 0x7f2d40116150 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 >> 172.21.15.73:0/1156835333 conn(0x7f2d40071d90 msgr2=0x7f2d40065720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 shutdown_connections 2024-02-10T06:40:29.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:29.410+0000 7f2d46c0b700 1 -- 172.21.15.73:0/1156835333 wait complete. 2024-02-10T06:40:30.012 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:29 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1156835333' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:40:30.037 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T06:40:30.037 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T06:40:30.038 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:40:30.047 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-10T06:40:30.048 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:40:30.097 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:40:30.098 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-10T06:40:30.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:29 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1156835333' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:40:30.353 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:30 smithi073 ceph-mon[29063]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:30 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:31.241 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:30 smithi181 ceph-mon[36721]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:31.242 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:30 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.963+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/3856986209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 msgr2=0x7fa3e80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.963+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/3856986209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e80fd420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc004940 tx=0x7fa3dc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/3856986209 shutdown_connections 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/3856986209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e80fd420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/3856986209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3e80fbe40 0x7fa3e80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/3856986209 >> 172.21.15.73:0/3856986209 conn(0x7fa3e80f7520 msgr2=0x7fa3e80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/3856986209 shutdown_connections 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/3856986209 wait complete. 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 Processor -- start 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- start start 2024-02-10T06:40:31.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3e80fbe40 0x7fa3e810d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e810de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e810e4f0 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3ecfe1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e810ed30 con 0x7fa3e80fbe40 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3e5d9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e810de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3e5d9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e810de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47126/0 (socket says 172.21.15.73:47126) 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.964+0000 7fa3e5d9b700 1 -- 172.21.15.73:0/2663375926 learned_addr learned my addr 172.21.15.73:0/2663375926 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3e5d9b700 1 -- 172.21.15.73:0/2663375926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3e80fbe40 msgr2=0x7fa3e810d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3e5d9b700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3e80fbe40 0x7fa3e810d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3e5d9b700 1 -- 172.21.15.73:0/2663375926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc005040 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3e5d9b700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e810de60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009fd0 tx=0x7fa3dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3dc00e040 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3dc00bcc0 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3dc01c630 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e810efb0 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.965+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e810f4a0 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.966+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e804bed0 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.966+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa3dc013980 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.967+0000 7fa3d6ffd700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3d00692b0 0x7fa3d006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:31.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.967+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa3dc088310 con 0x7fa3e80fcfb0 2024-02-10T06:40:31.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.967+0000 7fa3e659c700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3d00692b0 0x7fa3d006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:31.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.967+0000 7fa3e659c700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3d00692b0 0x7fa3d006b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006fd0 tx=0x7fa3d8006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:31.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:31.971+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa3dc058410 con 0x7fa3e80fcfb0 2024-02-10T06:40:32.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.157+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fa3e81049b0 con 0x7fa3e80fcfb0 2024-02-10T06:40:32.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.161+0000 7fa3d6ffd700 1 -- 172.21.15.73:0/2663375926 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fa3dc057fa0 con 0x7fa3e80fcfb0 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3d00692b0 msgr2=0x7fa3d006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3d00692b0 0x7fa3d006b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006fd0 tx=0x7fa3d8006db0 comp rx=0 tx=0).stop 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 msgr2=0x7fa3e810de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e810de60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009fd0 tx=0x7fa3dc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 shutdown_connections 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3d00692b0 0x7fa3d006b760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:32.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3e80fbe40 0x7fa3e810d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:32.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 --2- 172.21.15.73:0/2663375926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa3e80fcfb0 0x7fa3e810de60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:32.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 >> 172.21.15.73:0/2663375926 conn(0x7fa3e80f7520 msgr2=0x7fa3e8065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:32.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 shutdown_connections 2024-02-10T06:40:32.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:32.163+0000 7fa3ecfe1700 1 -- 172.21.15.73:0/2663375926 wait complete. 2024-02-10T06:40:32.811 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:32.811 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:32.811 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:32.811 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:32.811 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2663375926' entity='client.admin' 2024-02-10T06:40:32.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:32.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:32.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:32.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:32 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:32.836 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:40:32.845 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-10T06:40:32.845 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:40:32.893 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:40:32.893 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-10T06:40:33.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2663375926' entity='client.admin' 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:33.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:32 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:33.162 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:34.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- 172.21.15.73:0/2062637517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 msgr2=0x7f98dc0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:34.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 --2- 172.21.15.73:0/2062637517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc0fc250 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f98cc004940 tx=0x7f98cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:34.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- 172.21.15.73:0/2062637517 shutdown_connections 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 --2- 172.21.15.73:0/2062637517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98dc0fcfb0 0x7f98dc0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 --2- 172.21.15.73:0/2062637517 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc0fc250 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- 172.21.15.73:0/2062637517 >> 172.21.15.73:0/2062637517 conn(0x7f98dc0f7520 msgr2=0x7f98dc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- 172.21.15.73:0/2062637517 shutdown_connections 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- 172.21.15.73:0/2062637517 wait complete. 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 Processor -- start 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- start start 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98dc0fcfb0 0x7f98dc112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc066d80 con 0x7f98dc0fbe40 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.819+0000 7f98e0caa700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc066ec0 con 0x7f98dc0fcfb0 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc110670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47142/0 (socket says 172.21.15.73:47142) 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 -- 172.21.15.73:0/3260272755 learned_addr learned my addr 172.21.15.73:0/3260272755 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:34.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98d1d9b700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98dc0fcfb0 0x7f98dc112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 -- 172.21.15.73:0/3260272755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98dc0fcfb0 msgr2=0x7f98dc112bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98dc0fcfb0 0x7f98dc112bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 -- 172.21.15.73:0/3260272755 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc005040 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98da59c700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc110670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0093b0 tx=0x7f98cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98cc00e040 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc1131f0 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98cc00bcc0 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98cc01c630 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.820+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc1136d0 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.821+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc04bed0 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.822+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f98cc0139f0 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.822+0000 7f98d37fe700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f98c80692a0 0x7f98c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.822+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f98cc088350 con 0x7f98dc0fbe40 2024-02-10T06:40:34.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.822+0000 7f98d1d9b700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f98c80692a0 0x7f98c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:34.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.822+0000 7f98d1d9b700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f98c80692a0 0x7f98c806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98c4006fd0 tx=0x7f98c4008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:34.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:34.826+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f98cc0549c0 con 0x7f98dc0fbe40 2024-02-10T06:40:34.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:34 smithi073 ceph-mon[29063]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:35.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.022+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98dc101990 con 0x7f98c80692a0 2024-02-10T06:40:35.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.023+0000 7f98d37fe700 1 -- 172.21.15.73:0/3260272755 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f98dc101990 con 0x7f98c80692a0 2024-02-10T06:40:35.024 INFO:teuthology.orchestra.run.smithi073.stdout:Backend: cephadm 2024-02-10T06:40:35.024 INFO:teuthology.orchestra.run.smithi073.stdout:Available: Yes 2024-02-10T06:40:35.024 INFO:teuthology.orchestra.run.smithi073.stdout:Paused: No 2024-02-10T06:40:35.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.024+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f98c80692a0 msgr2=0x7f98c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.024+0000 7f98e0caa700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f98c80692a0 0x7f98c806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98c4006fd0 tx=0x7f98c4008040 comp rx=0 tx=0).stop 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.024+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 msgr2=0x7f98dc110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.024+0000 7f98e0caa700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc110670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0093b0 tx=0x7f98cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 shutdown_connections 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f98c80692a0 0x7f98c806b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f98dc0fbe40 0x7f98dc110670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 --2- 172.21.15.73:0/3260272755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98dc0fcfb0 0x7f98dc112bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 >> 172.21.15.73:0/3260272755 conn(0x7f98dc0f7520 msgr2=0x7f98dc100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 shutdown_connections 2024-02-10T06:40:35.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:35.025+0000 7f98e0caa700 1 -- 172.21.15.73:0/3260272755 wait complete. 2024-02-10T06:40:35.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:34 smithi181 ceph-mon[36721]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:35.760 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-10T06:40:36.015 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:36.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:36 smithi073 ceph-mon[29063]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:36.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:36 smithi073 ceph-mon[29063]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:37.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:36 smithi181 ceph-mon[36721]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:37.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:36 smithi181 ceph-mon[36721]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 -- 172.21.15.73:0/2113355118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 msgr2=0x7f30880fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 --2- 172.21.15.73:0/2113355118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f30880fc260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3080009ce0 tx=0x7f3080004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 -- 172.21.15.73:0/2113355118 shutdown_connections 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 --2- 172.21.15.73:0/2113355118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30880fcfc0 0x7f30880fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 --2- 172.21.15.73:0/2113355118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f30880fc260 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 -- 172.21.15.73:0/2113355118 >> 172.21.15.73:0/2113355118 conn(0x7f30880f7510 msgr2=0x7f30880f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 -- 172.21.15.73:0/2113355118 shutdown_connections 2024-02-10T06:40:37.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.792+0000 7f308e097700 1 -- 172.21.15.73:0/2113355118 wait complete. 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f308e097700 1 Processor -- start 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f308e097700 1 -- start start 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f308e097700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f308810dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f308e097700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30880fcfc0 0x7f308810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f308e097700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308810e5f0 con 0x7f30880fbe50 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f308e097700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308810ee90 con 0x7f30880fcfc0 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f30877fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f308810dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f30877fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f308810dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47174/0 (socket says 172.21.15.73:47174) 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f30877fe700 1 -- 172.21.15.73:0/628910437 learned_addr learned my addr 172.21.15.73:0/628910437 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.793+0000 7f3086ffd700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30880fcfc0 0x7f308810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f30877fe700 1 -- 172.21.15.73:0/628910437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30880fcfc0 msgr2=0x7f308810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f30877fe700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30880fcfc0 0x7f308810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:37.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f30877fe700 1 -- 172.21.15.73:0/628910437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3080009940 con 0x7f30880fbe50 2024-02-10T06:40:37.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f30877fe700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f308810dab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3080015040 tx=0x7f3080004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:37.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308000e040 con 0x7f30880fbe50 2024-02-10T06:40:37.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30800128e0 con 0x7f30880fbe50 2024-02-10T06:40:37.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308810f110 con 0x7f30880fbe50 2024-02-10T06:40:37.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308001c760 con 0x7f30880fbe50 2024-02-10T06:40:37.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.794+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308810f600 con 0x7f30880fbe50 2024-02-10T06:40:37.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.795+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308804bed0 con 0x7f30880fbe50 2024-02-10T06:40:37.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.795+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3080012a50 con 0x7f30880fbe50 2024-02-10T06:40:37.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.796+0000 7f306ffff700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30700692a0 0x7f307006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:37.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.796+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3080088b50 con 0x7f30880fbe50 2024-02-10T06:40:37.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.796+0000 7f3086ffd700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30700692a0 0x7f307006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:37.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.796+0000 7f3086ffd700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30700692a0 0x7f307006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f307c004f40 tx=0x7f307c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.800+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3080058bd0 con 0x7f30880fbe50 2024-02-10T06:40:37.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.990+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f308810fcc0 con 0x7f30700692a0 2024-02-10T06:40:37.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:37.998+0000 7f306ffff700 1 -- 172.21.15.73:0/628910437 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f308810fcc0 con 0x7f30700692a0 2024-02-10T06:40:37.999 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:40:37.999 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (3m) 2m ago 5m 27.7M - 0.25.0 c8568f914cd2 21a01485a256 2024-02-10T06:40:37.999 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (5m) 2m ago 5m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 26dabce1f0be 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi181 smithi181 running (4m) 71s ago 4m 6656k - 17.2.7-505-g72ce7c1d a2a0bd126c96 0ed4bbd7eaf3 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (3m) 2m ago 4m 80.5M - 9.4.7 954c08fa6188 9edb0a6c1f79 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.vjivfn smithi073 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-505-g72ce7c1d a2a0bd126c96 df67d3a17ded 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi181.lmidhc smithi181 *:8443,9283 running (4m) 71s ago 4m 432M - 17.2.7-505-g72ce7c1d a2a0bd126c96 0ede8e2f6c22 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (6m) 2m ago 6m 44.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 9743a7ae51ad 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi181 smithi181 running (4m) 71s ago 4m 40.4M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 335b19e9983e 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (5m) 2m ago 5m 16.2M - 1.5.0 0da6a335fe13 f5df22ebae55 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi181 smithi181 *:9100 running (4m) 71s ago 4m 16.2M - 1.5.0 0da6a335fe13 8b2c66b967b3 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (3m) 2m ago 3m 39.3M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 9218e1a18fa5 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (2m) 2m ago 2m 41.9M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 cffadaeafb44 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (2m) 2m ago 2m 41.2M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 c93fd57a90db 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi181 running (2m) 71s ago 2m 38.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 ebadff8f8025 2024-02-10T06:40:38.000 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi181 running (100s) 71s ago 100s 39.9M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 56929cb111f3 2024-02-10T06:40:38.001 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi181 running (76s) 71s ago 76s 12.7M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 85003a026df4 2024-02-10T06:40:38.001 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (3m) 2m ago 4m 38.0M - 2.43.0 a07b618ecd1d 2e631f28238b 2024-02-10T06:40:38.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30700692a0 msgr2=0x7f307006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:38.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30700692a0 0x7f307006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f307c004f40 tx=0x7f307c00b040 comp rx=0 tx=0).stop 2024-02-10T06:40:38.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 msgr2=0x7f308810dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f308810dab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3080015040 tx=0x7f3080004d10 comp rx=0 tx=0).stop 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 shutdown_connections 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30700692a0 0x7f307006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30880fbe50 0x7f308810dab0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 --2- 172.21.15.73:0/628910437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30880fcfc0 0x7f308810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 >> 172.21.15.73:0/628910437 conn(0x7f30880f7510 msgr2=0x7f3088065f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.000+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 shutdown_connections 2024-02-10T06:40:38.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:38.001+0000 7f308e097700 1 -- 172.21.15.73:0/628910437 wait complete. 2024-02-10T06:40:38.684 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-10T06:40:38.938 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:39.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:38 smithi073 ceph-mon[29063]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:39.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:38 smithi073 ceph-mon[29063]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:39.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:38 smithi181 ceph-mon[36721]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:39.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:38 smithi181 ceph-mon[36721]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:40.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- 172.21.15.73:0/3975670570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 msgr2=0x7f457c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:40.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 --2- 172.21.15.73:0/3975670570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c0fd420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4570004940 tx=0x7f4570009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:40.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- 172.21.15.73:0/3975670570 shutdown_connections 2024-02-10T06:40:40.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 --2- 172.21.15.73:0/3975670570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c0fd420 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:40.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 --2- 172.21.15.73:0/3975670570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 0x7f457c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- 172.21.15.73:0/3975670570 >> 172.21.15.73:0/3975670570 conn(0x7f457c0f7520 msgr2=0x7f457c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- 172.21.15.73:0/3975670570 shutdown_connections 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- 172.21.15.73:0/3975670570 wait complete. 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 Processor -- start 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- start start 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 0x7f457c110640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c112b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c066d10 con 0x7f457c0fcfb0 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.541+0000 7f4581619700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c066e50 con 0x7f457c0fbe40 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c112b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c112b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40584/0 (socket says 172.21.15.73:40584) 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 -- 172.21.15.73:0/1544782571 learned_addr learned my addr 172.21.15.73:0/1544782571 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457affd700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 0x7f457c110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:40.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 -- 172.21.15.73:0/1544782571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 msgr2=0x7f457c110640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 0x7f457c110640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 -- 172.21.15.73:0/1544782571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570005040 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457affd700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 0x7f457c110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f457a7fc700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c112b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f45700093b0 tx=0x7f4570009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457000e040 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c113190 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f457000bcc0 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457001c630 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.542+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c113680 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.543+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c04bed0 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.543+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f45700139f0 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.544+0000 7f45637fe700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f45640692a0 0x7f456406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.544+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4570088250 con 0x7f457c0fcfb0 2024-02-10T06:40:40.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.544+0000 7f457affd700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f45640692a0 0x7f456406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:40.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.544+0000 7f457affd700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f45640692a0 0x7f456406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f456c009fd0 tx=0x7f456c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:40.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.548+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f45700548c0 con 0x7f457c0fcfb0 2024-02-10T06:40:40.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.743+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f457c101990 con 0x7f45640692a0 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.747+0000 7f45637fe700 1 -- 172.21.15.73:0/1544782571 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f457c101990 con 0x7f45640692a0 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:crash 2/2 2m ago 6m * 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:mgr 2/2 2m ago 6m count:2 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:mon 2/2 2m ago 5m smithi073:172.21.15.73=smithi073;smithi181:172.21.15.181=smithi181;count:2 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-10T06:40:40.748 INFO:teuthology.orchestra.run.smithi073.stdout:osd 6 2m ago - 2024-02-10T06:40:40.749 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-10T06:40:40.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f45640692a0 msgr2=0x7f456406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:40.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f45640692a0 0x7f456406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f456c009fd0 tx=0x7f456c00b040 comp rx=0 tx=0).stop 2024-02-10T06:40:40.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 msgr2=0x7f457c112b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c112b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f45700093b0 tx=0x7f4570009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 shutdown_connections 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f45640692a0 0x7f456406b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f457c0fbe40 0x7f457c110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 --2- 172.21.15.73:0/1544782571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f457c0fcfb0 0x7f457c112b90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 >> 172.21.15.73:0/1544782571 conn(0x7f457c0f7520 msgr2=0x7f457c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.749+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 shutdown_connections 2024-02-10T06:40:40.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:40.750+0000 7f4581619700 1 -- 172.21.15.73:0/1544782571 wait complete. 2024-02-10T06:40:41.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:40 smithi181 ceph-mon[36721]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:41.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:40 smithi073 ceph-mon[29063]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:41.456 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-10T06:40:41.711 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:41.833 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:41 smithi073 ceph-mon[29063]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:41.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:41 smithi181 ceph-mon[36721]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:42.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:42 smithi073 ceph-mon[29063]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:42.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:43.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:42 smithi181 ceph-mon[36721]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:43.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:43.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 -- 172.21.15.73:0/1941863681 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fcef0 msgr2=0x7f91700fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:43.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 --2- 172.21.15.73:0/1941863681 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fcef0 0x7f91700fd320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9168004940 tx=0x7f9168009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 -- 172.21.15.73:0/1941863681 shutdown_connections 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 --2- 172.21.15.73:0/1941863681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91700fe0e0 0x7f91700fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 --2- 172.21.15.73:0/1941863681 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fcef0 0x7f91700fd320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 -- 172.21.15.73:0/1941863681 >> 172.21.15.73:0/1941863681 conn(0x7f91700f8620 msgr2=0x7f91700faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 -- 172.21.15.73:0/1941863681 shutdown_connections 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.358+0000 7f9177779700 1 -- 172.21.15.73:0/1941863681 wait complete. 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9177779700 1 Processor -- start 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9177779700 1 -- start start 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9177779700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91700fcef0 0x7f917010d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9177779700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 0x7f917010def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9177779700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917010e580 con 0x7f91700fe0e0 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9177779700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917010edc0 con 0x7f91700fcef0 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 0x7f917010def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:43.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 0x7f917010def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40594/0 (socket says 172.21.15.73:40594) 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 -- 172.21.15.73:0/2306445238 learned_addr learned my addr 172.21.15.73:0/2306445238 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 -- 172.21.15.73:0/2306445238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91700fcef0 msgr2=0x7f917010d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91700fcef0 0x7f917010d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 -- 172.21.15.73:0/2306445238 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168005040 con 0x7f91700fe0e0 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.359+0000 7f9174d14700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 0x7f917010def0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f916000b790 tx=0x7f9160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.360+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f916000f770 con 0x7f91700fe0e0 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.360+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917010f0a0 con 0x7f91700fe0e0 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.360+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f916000fda0 con 0x7f91700fe0e0 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.360+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f916000d310 con 0x7f91700fe0e0 2024-02-10T06:40:43.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.360+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917010f5e0 con 0x7f91700fe0e0 2024-02-10T06:40:43.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.361+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917004bed0 con 0x7f91700fe0e0 2024-02-10T06:40:43.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.361+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9160010420 con 0x7f91700fe0e0 2024-02-10T06:40:43.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.361+0000 7f9165ffb700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f915c0692a0 0x7f915c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:43.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.362+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9160086e70 con 0x7f91700fe0e0 2024-02-10T06:40:43.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.362+0000 7f9175515700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f915c0692a0 0x7f915c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:43.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.362+0000 7f9175515700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f915c0692a0 0x7f915c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9168004f90 tx=0x7f9168009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:43.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.366+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9160056f70 con 0x7f91700fe0e0 2024-02-10T06:40:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.562+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f917010fb00 con 0x7f915c0692a0 2024-02-10T06:40:43.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.563+0000 7f9165ffb700 1 -- 172.21.15.73:0/2306445238 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f917010fb00 con 0x7f915c0692a0 2024-02-10T06:40:43.564 INFO:teuthology.orchestra.run.smithi073.stdout:HOST ADDR LABELS STATUS 2024-02-10T06:40:43.564 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073 172.21.15.73 2024-02-10T06:40:43.564 INFO:teuthology.orchestra.run.smithi073.stdout:smithi181 172.21.15.181 2024-02-10T06:40:43.564 INFO:teuthology.orchestra.run.smithi073.stdout:2 hosts in cluster 2024-02-10T06:40:43.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f915c0692a0 msgr2=0x7f915c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:43.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f915c0692a0 0x7f915c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9168004f90 tx=0x7f9168009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:43.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 msgr2=0x7f917010def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:43.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 0x7f917010def0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f916000b790 tx=0x7f9160009250 comp rx=0 tx=0).stop 2024-02-10T06:40:43.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 shutdown_connections 2024-02-10T06:40:43.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f915c0692a0 0x7f915c06b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:43.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91700fcef0 0x7f917010d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:43.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 --2- 172.21.15.73:0/2306445238 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f91700fe0e0 0x7f917010def0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:43.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 >> 172.21.15.73:0/2306445238 conn(0x7f91700f8620 msgr2=0x7f91701013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:43.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 shutdown_connections 2024-02-10T06:40:43.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:43.565+0000 7f9177779700 1 -- 172.21.15.73:0/2306445238 wait complete. 2024-02-10T06:40:44.199 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-10T06:40:44.454 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:44.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:44 smithi073 ceph-mon[29063]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:44.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:44 smithi073 ceph-mon[29063]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:45.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:44 smithi181 ceph-mon[36721]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:45.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:44 smithi181 ceph-mon[36721]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:46.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.080+0000 7f02d7310700 1 -- 172.21.15.73:0/407142580 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 msgr2=0x7f02d00fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:46.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.080+0000 7f02d7310700 1 --2- 172.21.15.73:0/407142580 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d00fc8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02cc004940 tx=0x7f02cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:46.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- 172.21.15.73:0/407142580 shutdown_connections 2024-02-10T06:40:46.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 --2- 172.21.15.73:0/407142580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 0x7f02d00fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:46.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 --2- 172.21.15.73:0/407142580 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d00fc8c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- 172.21.15.73:0/407142580 >> 172.21.15.73:0/407142580 conn(0x7f02d00f6d20 msgr2=0x7f02d00f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- 172.21.15.73:0/407142580 shutdown_connections 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- 172.21.15.73:0/407142580 wait complete. 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 Processor -- start 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- start start 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d0110930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 0x7f02d0112e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00fbfe0 con 0x7f02d00fc4b0 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d7310700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00fc120 con 0x7f02d00fce00 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d50ac700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d0110930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d50ac700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d0110930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40610/0 (socket says 172.21.15.73:40610) 2024-02-10T06:40:46.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.081+0000 7f02d50ac700 1 -- 172.21.15.73:0/1255444228 learned_addr learned my addr 172.21.15.73:0/1255444228 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d48ab700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 0x7f02d0112e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d50ac700 1 -- 172.21.15.73:0/1255444228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 msgr2=0x7f02d0112e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d50ac700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 0x7f02d0112e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d50ac700 1 -- 172.21.15.73:0/1255444228 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc005040 con 0x7f02d00fc4b0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d48ab700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 0x7f02d0112e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d50ac700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d0110930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0093b0 tx=0x7f02cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02cc00e040 con 0x7f02d00fc4b0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02cc00bd90 con 0x7f02d00fc4b0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02cc01c700 con 0x7f02d00fc4b0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d0113500 con 0x7f02d00fc4b0 2024-02-10T06:40:46.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.082+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d0113a40 con 0x7f02d00fc4b0 2024-02-10T06:40:46.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.083+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d0106a00 con 0x7f02d00fc4b0 2024-02-10T06:40:46.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.084+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f02cc013b10 con 0x7f02d00fc4b0 2024-02-10T06:40:46.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.084+0000 7f02c5ffb700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f02bc0692a0 0x7f02bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:46.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.084+0000 7f02d48ab700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f02bc0692a0 0x7f02bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:46.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.084+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f02cc088240 con 0x7f02d00fc4b0 2024-02-10T06:40:46.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.084+0000 7f02d48ab700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f02bc0692a0 0x7f02bc06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f02c0006fd0 tx=0x7f02c0008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:46.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.088+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f02cc054950 con 0x7f02d00fc4b0 2024-02-10T06:40:46.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.281+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f02d0104a30 con 0x7f02bc0692a0 2024-02-10T06:40:46.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.282+0000 7f02c5ffb700 1 -- 172.21.15.73:0/1255444228 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f02d0104a30 con 0x7f02bc0692a0 2024-02-10T06:40:46.283 INFO:teuthology.orchestra.run.smithi073.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-10T06:40:46.283 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040115400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-10T06:40:46.283 INFO:teuthology.orchestra.run.smithi073.stdout:smithi181 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000E400BGN 372G No 76s ago Has a FileSystem, LVM detected 2024-02-10T06:40:46.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f02bc0692a0 msgr2=0x7f02bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f02bc0692a0 0x7f02bc06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f02c0006fd0 tx=0x7f02c0008040 comp rx=0 tx=0).stop 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 msgr2=0x7f02d0110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d0110930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0093b0 tx=0x7f02cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 shutdown_connections 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f02bc0692a0 0x7f02bc06b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f02d00fc4b0 0x7f02d0110930 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 --2- 172.21.15.73:0/1255444228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02d00fce00 0x7f02d0112e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.284+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 >> 172.21.15.73:0/1255444228 conn(0x7f02d00f6d20 msgr2=0x7f02d01005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.285+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 shutdown_connections 2024-02-10T06:40:46.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:46.285+0000 7f02d7310700 1 -- 172.21.15.73:0/1255444228 wait complete. 2024-02-10T06:40:47.046 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:46 smithi073 ceph-mon[29063]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:47.070 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:40:47.086 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:40:47.086 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-10T06:40:47.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:46 smithi181 ceph-mon[36721]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:47.340 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:48.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:47 smithi181 ceph-mon[36721]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:48.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:47 smithi073 ceph-mon[29063]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:48.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:48 smithi073 ceph-mon[29063]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:49.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:48 smithi181 ceph-mon[36721]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 -- 172.21.15.73:0/183083593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fe0e0 msgr2=0x7f202c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 --2- 172.21.15.73:0/183083593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fe0e0 0x7f202c0fe550 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2018004940 tx=0x7f2018009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 -- 172.21.15.73:0/183083593 shutdown_connections 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 --2- 172.21.15.73:0/183083593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fe0e0 0x7f202c0fe550 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 --2- 172.21.15.73:0/183083593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0fcef0 0x7f202c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 -- 172.21.15.73:0/183083593 >> 172.21.15.73:0/183083593 conn(0x7f202c0f8620 msgr2=0x7f202c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 -- 172.21.15.73:0/183083593 shutdown_connections 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.321+0000 7f203311e700 1 -- 172.21.15.73:0/183083593 wait complete. 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f203311e700 1 Processor -- start 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f203311e700 1 -- start start 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f203311e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 0x7f202c10d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:49.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f203311e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0fe0e0 0x7f202c10ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f203311e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c10e560 con 0x7f202c0fcef0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f203311e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c10eda0 con 0x7f202c0fe0e0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f2030eba700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 0x7f202c10d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f2030eba700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 0x7f202c10d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35016/0 (socket says 172.21.15.73:35016) 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.322+0000 7f2030eba700 1 -- 172.21.15.73:0/3648391804 learned_addr learned my addr 172.21.15.73:0/3648391804 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f202bfff700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0fe0e0 0x7f202c10ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f2030eba700 1 -- 172.21.15.73:0/3648391804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0fe0e0 msgr2=0x7f202c10ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f2030eba700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0fe0e0 0x7f202c10ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f2030eba700 1 -- 172.21.15.73:0/3648391804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2018005040 con 0x7f202c0fcef0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f2030eba700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 0x7f202c10d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f202000b790 tx=0x7f2020009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f202000f6f0 con 0x7f202c0fcef0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f202000fd20 con 0x7f202c0fcef0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c10f080 con 0x7f202c0fcef0 2024-02-10T06:40:49.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20200103a0 con 0x7f202c0fcef0 2024-02-10T06:40:49.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.323+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c10f5c0 con 0x7f202c0fcef0 2024-02-10T06:40:49.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.324+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202c04bed0 con 0x7f202c0fcef0 2024-02-10T06:40:49.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.324+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f202000f850 con 0x7f202c0fcef0 2024-02-10T06:40:49.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.325+0000 7f20297fa700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f201c0692a0 0x7f201c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:49.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.325+0000 7f202bfff700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f201c0692a0 0x7f201c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:49.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.325+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2020086cf0 con 0x7f202c0fcef0 2024-02-10T06:40:49.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.325+0000 7f202bfff700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f201c0692a0 0x7f201c06b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2018004f90 tx=0x7f2018009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:49.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.329+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2020056d70 con 0x7f202c0fcef0 2024-02-10T06:40:49.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:49.540+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f202c10f210 con 0x7f201c0692a0 2024-02-10T06:40:50.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:50 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-10T06:40:50.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:50 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-10T06:40:50.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.857+0000 7f20297fa700 1 -- 172.21.15.73:0/3648391804 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f202c10f210 con 0x7f201c0692a0 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f201c0692a0 msgr2=0x7f201c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f201c0692a0 0x7f201c06b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2018004f90 tx=0x7f2018009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 msgr2=0x7f202c10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 0x7f202c10d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f202000b790 tx=0x7f2020009220 comp rx=0 tx=0).stop 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 shutdown_connections 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f201c0692a0 0x7f201c06b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:50.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f202c0fcef0 0x7f202c10d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:50.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 --2- 172.21.15.73:0/3648391804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0fe0e0 0x7f202c10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:50.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 >> 172.21.15.73:0/3648391804 conn(0x7f202c0f8620 msgr2=0x7f202c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:50.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 shutdown_connections 2024-02-10T06:40:50.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:50.860+0000 7f203311e700 1 -- 172.21.15.73:0/3648391804 wait complete. 2024-02-10T06:40:51.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:40:50.834+0000 7f6f4d35f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-10T06:40:51.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: fsmap cephfs:0 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: Saving service mds.cephfs spec with placement count:4 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.nuvejb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.nuvejb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:51.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:51 smithi073 ceph-mon[29063]: Deploying daemon mds.cephfs.smithi073.nuvejb on smithi073 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T06:40:51.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: fsmap cephfs:0 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: Saving service mds.cephfs spec with placement count:4 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.nuvejb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.nuvejb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:51.342 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:51 smithi181 ceph-mon[36721]: Deploying daemon mds.cephfs.smithi073.nuvejb on smithi073 2024-02-10T06:40:51.535 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-10T06:40:52.763 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:53.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:52 smithi073 ceph-mon[29063]: pgmap v179: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:53.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:52 smithi073 ceph-mon[29063]: osdmap e45: 6 total, 6 up, 6 in 2024-02-10T06:40:53.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:52 smithi181 ceph-mon[36721]: pgmap v179: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:53.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:52 smithi181 ceph-mon[36721]: osdmap e45: 6 total, 6 up, 6 in 2024-02-10T06:40:54.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:53 smithi073 ceph-mon[29063]: osdmap e46: 6 total, 6 up, 6 in 2024-02-10T06:40:54.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:53 smithi181 ceph-mon[36721]: osdmap e46: 6 total, 6 up, 6 in 2024-02-10T06:40:54.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 -- 172.21.15.73:0/1256836320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 msgr2=0x7f31900ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 --2- 172.21.15.73:0/1256836320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f31900ffb10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3180004940 tx=0x7f3180009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 -- 172.21.15.73:0/1256836320 shutdown_connections 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 --2- 172.21.15.73:0/1256836320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31901000e0 0x7f3190102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 --2- 172.21.15.73:0/1256836320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f31900ffb10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 -- 172.21.15.73:0/1256836320 >> 172.21.15.73:0/1256836320 conn(0x7f31900f7520 msgr2=0x7f31900f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 -- 172.21.15.73:0/1256836320 shutdown_connections 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.529+0000 7f31980e5700 1 -- 172.21.15.73:0/1256836320 wait complete. 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f31980e5700 1 Processor -- start 2024-02-10T06:40:54.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f31980e5700 1 -- start start 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f31980e5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f319010d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f31980e5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31901000e0 0x7f319010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f31980e5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319010e480 con 0x7f3190066c50 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f31980e5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319010ed20 con 0x7f31901000e0 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f3195e81700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f319010d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f3195e81700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f319010d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35022/0 (socket says 172.21.15.73:35022) 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.530+0000 7f3195e81700 1 -- 172.21.15.73:0/3969367792 learned_addr learned my addr 172.21.15.73:0/3969367792 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:54.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f3195e81700 1 -- 172.21.15.73:0/3969367792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31901000e0 msgr2=0x7f319010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f3195e81700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31901000e0 0x7f319010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f3195e81700 1 -- 172.21.15.73:0/3969367792 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3180005040 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f3195e81700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f319010d940 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f318000b3b0 tx=0x7f3180009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318000e040 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f318000bd90 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319010efa0 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318001c630 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.531+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319010f4e0 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.532+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3190107a70 con 0x7f3190066c50 2024-02-10T06:40:54.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.532+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f31800139f0 con 0x7f3190066c50 2024-02-10T06:40:54.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.533+0000 7f31867fc700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f317c0692b0 0x7f317c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:54.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.533+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f31800885e0 con 0x7f3190066c50 2024-02-10T06:40:54.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.533+0000 7f3195680700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f317c0692b0 0x7f317c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:54.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.533+0000 7f3195680700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f317c0692b0 0x7f317c06b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3190066990 tx=0x7f318c009ce0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:54.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.537+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3180058440 con 0x7f3190066c50 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.774+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3190105a30 con 0x7f3190066c50 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.774+0000 7f31867fc700 1 -- 172.21.15.73:0/3969367792 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f3180057fd0 con 0x7f3190066c50 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stdout:e2 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stdout:legacy client fscid: 1 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:40:54.775 INFO:teuthology.orchestra.run.smithi073.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:fs_name cephfs 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 2 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-02-10T06:40:50.835658+0000 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:modified 2024-02-10T06:40:50.835721+0000 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:tableserver 0 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:root 0 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:session_timeout 60 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:session_autoclose 300 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:max_file_size 1099511627776 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:required_client_features {} 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure 0 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure_osd_epoch 0 2024-02-10T06:40:54.776 INFO:teuthology.orchestra.run.smithi073.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:max_mds 1 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:in 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:up {} 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:failed 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:damaged 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:stopped 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:data_pools [3] 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:metadata_pool 2 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:inline_data disabled 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:balancer 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout:standby_count_wanted 0 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:40:54.777 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:40:54.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.776+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f317c0692b0 msgr2=0x7f317c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:54.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.776+0000 7f31980e5700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f317c0692b0 0x7f317c06b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3190066990 tx=0x7f318c009ce0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.776+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 msgr2=0x7f319010d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:54.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.776+0000 7f31980e5700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f319010d940 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f318000b3b0 tx=0x7f3180009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:54.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.776+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 shutdown_connections 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.776+0000 7f31980e5700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f317c0692b0 0x7f317c06b760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.777+0000 7f31980e5700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3190066c50 0x7f319010d940 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.777+0000 7f31980e5700 1 --2- 172.21.15.73:0/3969367792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31901000e0 0x7f319010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.777+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 >> 172.21.15.73:0/3969367792 conn(0x7f31900f7520 msgr2=0x7f31900f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.777+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 shutdown_connections 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:54.777+0000 7f31980e5700 1 -- 172.21.15.73:0/3969367792 wait complete. 2024-02-10T06:40:54.779 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 2 2024-02-10T06:40:55.028 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:54 smithi073 ceph-mon[29063]: pgmap v182: 65 pgs: 9 creating+peering, 25 unknown, 31 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:55.028 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:54 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3969367792' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:40:55.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:54 smithi181 ceph-mon[36721]: pgmap v182: 65 pgs: 9 creating+peering, 25 unknown, 31 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:55.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:54 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3969367792' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:40:55.683 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:40:55.693 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:40:55.693 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-02-10T06:40:55.957 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:40:56.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.cuebex", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.cuebex", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: Deploying daemon mds.cephfs.smithi181.cuebex on smithi181 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: pgmap v183: 65 pgs: 9 creating+peering, 6 unknown, 50 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.73:6826/3433504765,v1:172.21.15.73:6827/3433504765] up:boot 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: daemon mds.cephfs.smithi073.nuvejb assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: Cluster is now healthy 2024-02-10T06:40:56.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: fsmap cephfs:0 1 up:standby 2024-02-10T06:40:56.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.nuvejb"}]: dispatch 2024-02-10T06:40:56.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:creating} 2024-02-10T06:40:56.092 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:56 smithi181 ceph-mon[36721]: daemon mds.cephfs.smithi073.nuvejb is now active in filesystem cephfs as rank 0 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.cuebex", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.cuebex", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: Deploying daemon mds.cephfs.smithi181.cuebex on smithi181 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: pgmap v183: 65 pgs: 9 creating+peering, 6 unknown, 50 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.73:6826/3433504765,v1:172.21.15.73:6827/3433504765] up:boot 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: daemon mds.cephfs.smithi073.nuvejb assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T06:40:56.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: Cluster is now healthy 2024-02-10T06:40:56.344 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: fsmap cephfs:0 1 up:standby 2024-02-10T06:40:56.344 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.nuvejb"}]: dispatch 2024-02-10T06:40:56.344 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:creating} 2024-02-10T06:40:56.344 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:56 smithi073 ceph-mon[29063]: daemon mds.cephfs.smithi073.nuvejb is now active in filesystem cephfs as rank 0 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 -- 172.21.15.73:0/2574511798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fe2b0 msgr2=0x7f26dc0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/2574511798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fe2b0 0x7f26dc0fe700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f26cc004940 tx=0x7f26cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 -- 172.21.15.73:0/2574511798 shutdown_connections 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/2574511798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fe2b0 0x7f26dc0fe700 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/2574511798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26dc0fb220 0x7f26dc0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 -- 172.21.15.73:0/2574511798 >> 172.21.15.73:0/2574511798 conn(0x7f26dc0f6ca0 msgr2=0x7f26dc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.579+0000 7f26e3fa4700 1 -- 172.21.15.73:0/2574511798 shutdown_connections 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 -- 172.21.15.73:0/2574511798 wait complete. 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 Processor -- start 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 -- start start 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 0x7f26dc10d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:57.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26dc0fe2b0 0x7f26dc10de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc10e490 con 0x7f26dc0fb220 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e3fa4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc10ed30 con 0x7f26dc0fe2b0 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e1d40700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 0x7f26dc10d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e1d40700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 0x7f26dc10d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35064/0 (socket says 172.21.15.73:35064) 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.580+0000 7f26e1d40700 1 -- 172.21.15.73:0/764264386 learned_addr learned my addr 172.21.15.73:0/764264386 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e153f700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26dc0fe2b0 0x7f26dc10de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e1d40700 1 -- 172.21.15.73:0/764264386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26dc0fe2b0 msgr2=0x7f26dc10de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e1d40700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26dc0fe2b0 0x7f26dc10de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e1d40700 1 -- 172.21.15.73:0/764264386 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc005040 con 0x7f26dc0fb220 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e1d40700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 0x7f26dc10d950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f26d8002a80 tx=0x7f26d80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d8007cc0 con 0x7f26dc0fb220 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d800f040 con 0x7f26dc0fb220 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc10f010 con 0x7f26dc0fb220 2024-02-10T06:40:57.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d8009630 con 0x7f26dc0fb220 2024-02-10T06:40:57.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.581+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc10f550 con 0x7f26dc0fb220 2024-02-10T06:40:57.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.582+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc04bed0 con 0x7f26dc0fb220 2024-02-10T06:40:57.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.583+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f26d8009790 con 0x7f26dc0fb220 2024-02-10T06:40:57.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.583+0000 7f26d27fc700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26c80692a0 0x7f26c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:40:57.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.583+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f26d80869d0 con 0x7f26dc0fb220 2024-02-10T06:40:57.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.583+0000 7f26e153f700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26c80692a0 0x7f26c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:40:57.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.584+0000 7f26e153f700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26c80692a0 0x7f26c806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f26cc004f90 tx=0x7f26cc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:40:57.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.588+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f26d804cbd0 con 0x7f26dc0fb220 2024-02-10T06:40:57.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.831+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f26dc10f900 con 0x7f26dc0fb220 2024-02-10T06:40:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.872+0000 7f26d27fc700 1 -- 172.21.15.73:0/764264386 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f26d8056730 con 0x7f26dc0fb220 2024-02-10T06:40:57.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26c80692a0 msgr2=0x7f26c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26c80692a0 0x7f26c806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f26cc004f90 tx=0x7f26cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 msgr2=0x7f26dc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 0x7f26dc10d950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f26d8002a80 tx=0x7f26d80046b0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 shutdown_connections 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f26c80692a0 0x7f26c806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26dc0fb220 0x7f26dc10d950 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 --2- 172.21.15.73:0/764264386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26dc0fe2b0 0x7f26dc10de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.874+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 >> 172.21.15.73:0/764264386 conn(0x7f26dc0f6ca0 msgr2=0x7f26dc1014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.875+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 shutdown_connections 2024-02-10T06:40:57.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:40:57.875+0000 7f26e3fa4700 1 -- 172.21.15.73:0/764264386 wait complete. 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.73:6826/3433504765,v1:172.21.15.73:6827/3433504765] up:active 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.ykxzap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:40:58.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.ykxzap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:40:58.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:58.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:57 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/764264386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-10T06:40:58.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.73:6826/3433504765,v1:172.21.15.73:6827/3433504765] up:active 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.ykxzap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi073.ykxzap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:58.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:57 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/764264386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: Deploying daemon mds.cephfs.smithi073.ykxzap on smithi073 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:boot 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/764264386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 1 up:standby 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi181.cuebex"}]: dispatch 2024-02-10T06:40:59.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:40:58 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 1 up:standby 2024-02-10T06:40:59.125 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-02-10T06:40:59.125 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: Deploying daemon mds.cephfs.smithi073.ykxzap on smithi073 2024-02-10T06:40:59.125 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:boot 2024-02-10T06:40:59.125 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/764264386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-10T06:40:59.125 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 1 up:standby 2024-02-10T06:40:59.126 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi181.cuebex"}]: dispatch 2024-02-10T06:40:59.126 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:40:58 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 1 up:standby 2024-02-10T06:40:59.157 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:40:59.166 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:40:59.166 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-02-10T06:40:59.766 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:00.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:00 smithi073 ceph-mon[29063]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:41:01.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:00 smithi181 ceph-mon[36721]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 -- 172.21.15.73:0/286540402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fe0e0 msgr2=0x7fb1480fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 --2- 172.21.15.73:0/286540402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fe0e0 0x7fb1480fe550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb13c004940 tx=0x7fb13c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 -- 172.21.15.73:0/286540402 shutdown_connections 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 --2- 172.21.15.73:0/286540402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fe0e0 0x7fb1480fe550 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 --2- 172.21.15.73:0/286540402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1480fcef0 0x7fb1480fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 -- 172.21.15.73:0/286540402 >> 172.21.15.73:0/286540402 conn(0x7fb1480f8620 msgr2=0x7fb1480faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 -- 172.21.15.73:0/286540402 shutdown_connections 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.448+0000 7fb14df8f700 1 -- 172.21.15.73:0/286540402 wait complete. 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb14df8f700 1 Processor -- start 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb14df8f700 1 -- start start 2024-02-10T06:41:01.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb14df8f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 0x7fb148071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb14df8f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1480fe0e0 0x7fb1480702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb14df8f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148070970 con 0x7fb1480fcef0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb14df8f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148070ae0 con 0x7fb1480fe0e0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb1477fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 0x7fb148071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb1477fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 0x7fb148071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42206/0 (socket says 172.21.15.73:42206) 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb1477fe700 1 -- 172.21.15.73:0/41873839 learned_addr learned my addr 172.21.15.73:0/41873839 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb146ffd700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1480fe0e0 0x7fb1480702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb1477fe700 1 -- 172.21.15.73:0/41873839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1480fe0e0 msgr2=0x7fb1480702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb1477fe700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1480fe0e0 0x7fb1480702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.449+0000 7fb1477fe700 1 -- 172.21.15.73:0/41873839 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c005040 con 0x7fb1480fcef0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.450+0000 7fb1477fe700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 0x7fb148071e60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb13800b790 tx=0x7fb138009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.450+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13800f6f0 con 0x7fb1480fcef0 2024-02-10T06:41:01.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.450+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb148070dc0 con 0x7fb1480fcef0 2024-02-10T06:41:01.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.450+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb13800fd20 con 0x7fb1480fcef0 2024-02-10T06:41:01.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.450+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1380103a0 con 0x7fb1480fcef0 2024-02-10T06:41:01.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.450+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb148071140 con 0x7fb1480fcef0 2024-02-10T06:41:01.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.451+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14804bed0 con 0x7fb1480fcef0 2024-02-10T06:41:01.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.451+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb138011400 con 0x7fb1480fcef0 2024-02-10T06:41:01.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.452+0000 7fb12ffff700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb1300692a0 0x7fb13006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:01.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.452+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb138087030 con 0x7fb1480fcef0 2024-02-10T06:41:01.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.452+0000 7fb146ffd700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb1300692a0 0x7fb13006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:01.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.452+0000 7fb146ffd700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb1300692a0 0x7fb13006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb13c004f90 tx=0x7fb13c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:01.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.456+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb138056d70 con 0x7fb1480fcef0 2024-02-10T06:41:01.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:01.688+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb148105ad0 con 0x7fb1480fcef0 2024-02-10T06:41:02.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.633+0000 7fb12ffff700 1 -- 172.21.15.73:0/41873839 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb138056900 con 0x7fb1480fcef0 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb1300692a0 msgr2=0x7fb13006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb1300692a0 0x7fb13006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb13c004f90 tx=0x7fb13c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 msgr2=0x7fb148071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 0x7fb148071e60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb13800b790 tx=0x7fb138009220 comp rx=0 tx=0).stop 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 shutdown_connections 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb1300692a0 0x7fb13006b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:02.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1480fcef0 0x7fb148071e60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:02.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 --2- 172.21.15.73:0/41873839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1480fe0e0 0x7fb1480702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:02.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 >> 172.21.15.73:0/41873839 conn(0x7fb1480f8620 msgr2=0x7fb1481013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:02.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 shutdown_connections 2024-02-10T06:41:02.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:02.635+0000 7fb14df8f700 1 -- 172.21.15.73:0/41873839 wait complete. 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/41873839' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.otunfp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.otunfp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:02.647 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:02 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/41873839' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.otunfp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi181.otunfp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:03.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:02 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:03.299 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:41:03.309 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:41:03.309 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-10T06:41:03.567 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:03.917 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:03 smithi181 ceph-mon[36721]: Deploying daemon mds.cephfs.smithi181.otunfp on smithi181 2024-02-10T06:41:03.917 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:03 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/41873839' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-10T06:41:03.917 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:03 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] up:boot 2024-02-10T06:41:03.917 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:03 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 2 up:standby 2024-02-10T06:41:03.917 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:03 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.ykxzap"}]: dispatch 2024-02-10T06:41:03.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:03 smithi073 ceph-mon[29063]: Deploying daemon mds.cephfs.smithi181.otunfp on smithi181 2024-02-10T06:41:03.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:03 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/41873839' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-10T06:41:03.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:03 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] up:boot 2024-02-10T06:41:03.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:03 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 2 up:standby 2024-02-10T06:41:03.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:03 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.ykxzap"}]: dispatch 2024-02-10T06:41:04.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:04 smithi181 ceph-mon[36721]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-02-10T06:41:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:04 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.922 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:04 smithi073 ceph-mon[29063]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-02-10T06:41:04.922 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.922 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.922 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:04.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:04 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:05.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.270+0000 7fadebe4f700 1 -- 172.21.15.73:0/3408673283 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 msgr2=0x7fade40ffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.270+0000 7fadebe4f700 1 --2- 172.21.15.73:0/3408673283 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade40ffb50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fade0004940 tx=0x7fade0009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- 172.21.15.73:0/3408673283 shutdown_connections 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 --2- 172.21.15.73:0/3408673283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade4100120 0x7fade4102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 --2- 172.21.15.73:0/3408673283 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade40ffb50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- 172.21.15.73:0/3408673283 >> 172.21.15.73:0/3408673283 conn(0x7fade40f74e0 msgr2=0x7fade40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- 172.21.15.73:0/3408673283 shutdown_connections 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- 172.21.15.73:0/3408673283 wait complete. 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 Processor -- start 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- start start 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade4100120 0x7fade410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:05.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade410e430 con 0x7fade4066380 2024-02-10T06:41:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fadebe4f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade410ecd0 con 0x7fade4100120 2024-02-10T06:41:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fade9beb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fade9beb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade410d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42254/0 (socket says 172.21.15.73:42254) 2024-02-10T06:41:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.271+0000 7fade9beb700 1 -- 172.21.15.73:0/2297797083 learned_addr learned my addr 172.21.15.73:0/2297797083 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:05.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.272+0000 7fade93ea700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade4100120 0x7fade410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.272+0000 7fade9beb700 1 -- 172.21.15.73:0/2297797083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade4100120 msgr2=0x7fade410de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.272+0000 7fade9beb700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade4100120 0x7fade410de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.272+0000 7fade9beb700 1 -- 172.21.15.73:0/2297797083 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade0005040 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.272+0000 7fade9beb700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade410d8f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fade0000c00 tx=0x7fade0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.272+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fade0021070 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.273+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fade001c400 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.273+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fade000e070 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.273+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade410ef50 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.273+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade410f430 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.274+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade404bed0 con 0x7fade4066380 2024-02-10T06:41:05.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.274+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fade001ca90 con 0x7fade4066380 2024-02-10T06:41:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.275+0000 7fadda7fc700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fadd00692a0 0x7fadd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.275+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fade0088290 con 0x7fade4066380 2024-02-10T06:41:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.275+0000 7fade93ea700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fadd00692a0 0x7fadd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:05.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.275+0000 7fade93ea700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fadd00692a0 0x7fadd006b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009b80 tx=0x7fadd400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:05.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.279+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fade00580f0 con 0x7fade4066380 2024-02-10T06:41:05.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.534+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fade410f950 con 0x7fade4066380 2024-02-10T06:41:05.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.647+0000 7fadda7fc700 1 -- 172.21.15.73:0/2297797083 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7fade0057c80 con 0x7fade4066380 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fadd00692a0 msgr2=0x7fadd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fadd00692a0 0x7fadd006b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009b80 tx=0x7fadd400b040 comp rx=0 tx=0).stop 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 msgr2=0x7fade410d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade410d8f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fade0000c00 tx=0x7fade0009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 shutdown_connections 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fadd00692a0 0x7fadd006b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fade4066380 0x7fade410d8f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 --2- 172.21.15.73:0/2297797083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade4100120 0x7fade410de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:05.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.650+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 >> 172.21.15.73:0/2297797083 conn(0x7fade40f74e0 msgr2=0x7fade40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:05.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.651+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 shutdown_connections 2024-02-10T06:41:05.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:05.651+0000 7fadebe4f700 1 -- 172.21.15.73:0/2297797083 wait complete. 2024-02-10T06:41:05.652 INFO:teuthology.orchestra.run.smithi073.stderr:inline data enabled 2024-02-10T06:41:05.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:05 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] up:boot 2024-02-10T06:41:05.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:05 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 3 up:standby 2024-02-10T06:41:05.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi181.otunfp"}]: dispatch 2024-02-10T06:41:05.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:05 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:05.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:05 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2297797083' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:41:05.949 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:05 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] up:boot 2024-02-10T06:41:05.949 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:05 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 3 up:standby 2024-02-10T06:41:05.949 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi181.otunfp"}]: dispatch 2024-02-10T06:41:05.949 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:05 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:05.949 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:05 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2297797083' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:41:06.659 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:41:06.659 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-10T06:41:06.659 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:standby 2024-02-10T06:41:06.659 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2297797083' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:41:06.659 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: Dropping low affinity active daemon mds.cephfs.smithi073.nuvejb in favor of higher affinity standby. 2024-02-10T06:41:06.660 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: Replacing daemon mds.cephfs.smithi073.nuvejb as rank 0 with standby daemon mds.cephfs.smithi181.cuebex 2024-02-10T06:41:06.660 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:41:06.660 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 3 up:standby 2024-02-10T06:41:06.660 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: osdmap e47: 6 total, 6 up, 6 in 2024-02-10T06:41:06.660 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:06 smithi073 ceph-mon[29063]: fsmap cephfs:1/1 {0=cephfs.smithi181.cuebex=up:replay} 2 up:standby 2024-02-10T06:41:06.687 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:41:06.697 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:41:06.697 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-10T06:41:07.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:41:07.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:standby 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2297797083' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: Dropping low affinity active daemon mds.cephfs.smithi073.nuvejb in favor of higher affinity standby. 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: Replacing daemon mds.cephfs.smithi073.nuvejb as rank 0 with standby daemon mds.cephfs.smithi181.cuebex 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi073.nuvejb=up:active} 3 up:standby 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: osdmap e47: 6 total, 6 up, 6 in 2024-02-10T06:41:07.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:06 smithi181 ceph-mon[36721]: fsmap cephfs:1/1 {0=cephfs.smithi181.cuebex=up:replay} 2 up:standby 2024-02-10T06:41:07.965 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:07.998 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:reconnect 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.73:6826/2661433887,v1:172.21.15.73:6827/2661433887] up:boot 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] up:standby 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: fsmap cephfs:1/1 {0=cephfs.smithi181.cuebex=up:reconnect} 3 up:standby 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.nuvejb"}]: dispatch 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:07.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:07 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:08.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:reconnect 2024-02-10T06:41:08.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.73:6826/2661433887,v1:172.21.15.73:6827/2661433887] up:boot 2024-02-10T06:41:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] up:standby 2024-02-10T06:41:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: fsmap cephfs:1/1 {0=cephfs.smithi181.cuebex=up:reconnect} 3 up:standby 2024-02-10T06:41:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi073.nuvejb"}]: dispatch 2024-02-10T06:41:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:08.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:07 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 409 B/s wr, 2 op/s 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:rejoin 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: fsmap cephfs:1/1 {0=cephfs.smithi181.cuebex=up:rejoin} 3 up:standby 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: daemon mds.cephfs.smithi181.cuebex is now active in filesystem cephfs as rank 0 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:active 2024-02-10T06:41:08.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: mds.? [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] up:standby 2024-02-10T06:41:08.697 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:08 smithi073 ceph-mon[29063]: fsmap cephfs:1 {0=cephfs.smithi181.cuebex=up:active} 3 up:standby 2024-02-10T06:41:09.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 409 B/s wr, 2 op/s 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:rejoin 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: fsmap cephfs:1/1 {0=cephfs.smithi181.cuebex=up:rejoin} 3 up:standby 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: daemon mds.cephfs.smithi181.cuebex is now active in filesystem cephfs as rank 0 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] up:active 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: mds.? [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] up:standby 2024-02-10T06:41:09.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:08 smithi181 ceph-mon[36721]: fsmap cephfs:1 {0=cephfs.smithi181.cuebex=up:active} 3 up:standby 2024-02-10T06:41:10.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.580+0000 7f3084fb2700 1 -- 172.21.15.73:0/3729991382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fcef0 msgr2=0x7f30800fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:10.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.580+0000 7f3084fb2700 1 --2- 172.21.15.73:0/3729991382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fcef0 0x7f30800fd320 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3070004940 tx=0x7f3070009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:10.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- 172.21.15.73:0/3729991382 shutdown_connections 2024-02-10T06:41:10.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 --2- 172.21.15.73:0/3729991382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fe0e0 0x7f30800fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:10.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 --2- 172.21.15.73:0/3729991382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fcef0 0x7f30800fd320 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- 172.21.15.73:0/3729991382 >> 172.21.15.73:0/3729991382 conn(0x7f30800f8620 msgr2=0x7f30800faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- 172.21.15.73:0/3729991382 shutdown_connections 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- 172.21.15.73:0/3729991382 wait complete. 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 Processor -- start 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- start start 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fcef0 0x7f3080071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 0x7f30800702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30800708e0 con 0x7f30800fe0e0 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f3084fb2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080070a50 con 0x7f30800fcef0 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f307dd9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 0x7f30800702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f307dd9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 0x7f30800702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53060/0 (socket says 172.21.15.73:53060) 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.581+0000 7f307dd9b700 1 -- 172.21.15.73:0/1533284642 learned_addr learned my addr 172.21.15.73:0/1533284642 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f307e59c700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fcef0 0x7f3080071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:10.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f307dd9b700 1 -- 172.21.15.73:0/1533284642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fcef0 msgr2=0x7f3080071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f307dd9b700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fcef0 0x7f3080071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f307dd9b700 1 -- 172.21.15.73:0/1533284642 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3070005040 con 0x7f30800fe0e0 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f307e59c700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fcef0 0x7f3080071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f307dd9b700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 0x7f30800702e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f307400c830 tx=0x7f3074004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f307400a760 con 0x7f30800fe0e0 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f307400ad90 con 0x7f30800fe0e0 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30740113a0 con 0x7f30800fe0e0 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3080070d30 con 0x7f30800fe0e0 2024-02-10T06:41:10.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.582+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30801159a0 con 0x7f30800fe0e0 2024-02-10T06:41:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.583+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308004bed0 con 0x7f30800fe0e0 2024-02-10T06:41:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.583+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3074012400 con 0x7f30800fe0e0 2024-02-10T06:41:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.584+0000 7f306effd700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30680692a0 0x7f306806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.584+0000 7f307e59c700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30680692a0 0x7f306806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.584+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3074087190 con 0x7f30800fe0e0 2024-02-10T06:41:10.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.584+0000 7f307e59c700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30680692a0 0x7f306806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f307000b3b0 tx=0x7f3070009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:10.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.588+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3074056fa0 con 0x7f30800fe0e0 2024-02-10T06:41:10.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.824+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3080115bf0 con 0x7f30800fe0e0 2024-02-10T06:41:10.826 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:10 smithi073 ceph-mon[29063]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 409 B/s wr, 2 op/s 2024-02-10T06:41:10.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.825+0000 7f306effd700 1 -- 172.21.15.73:0/1533284642 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1811 (secure 0 0 0) 0x7f3080115bf0 con 0x7f30800fe0e0 2024-02-10T06:41:10.826 INFO:teuthology.orchestra.run.smithi073.stdout:e14 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:legacy client fscid: 1 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:fs_name cephfs 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 14 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-02-10T06:40:50.835658+0000 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:modified 2024-02-10T06:41:08.652904+0000 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:tableserver 0 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:root 0 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:session_timeout 60 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:session_autoclose 300 2024-02-10T06:41:10.827 INFO:teuthology.orchestra.run.smithi073.stdout:max_file_size 1099511627776 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:required_client_features {} 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure 0 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure_osd_epoch 47 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:max_mds 1 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:in 0 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:up {0=24237} 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:failed 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:damaged 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:stopped 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:data_pools [3] 2024-02-10T06:41:10.828 INFO:teuthology.orchestra.run.smithi073.stdout:metadata_pool 2 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:inline_data enabled 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:balancer 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:standby_count_wanted 1 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi181.cuebex{0:24237} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:Standby daemons: 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi073.ykxzap{-1:14476} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi181.otunfp{-1:24249} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:10.829 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi073.nuvejb{-1:24257} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.73:6826/2661433887,v1:172.21.15.73:6827/2661433887] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30680692a0 msgr2=0x7f306806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30680692a0 0x7f306806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f307000b3b0 tx=0x7f3070009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 msgr2=0x7f30800702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 0x7f30800702e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f307400c830 tx=0x7f3074004470 comp rx=0 tx=0).stop 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 shutdown_connections 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f30680692a0 0x7f306806b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30800fcef0 0x7f3080071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:10.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 --2- 172.21.15.73:0/1533284642 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30800fe0e0 0x7f30800702e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:10.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 >> 172.21.15.73:0/1533284642 conn(0x7f30800f8620 msgr2=0x7f30801013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:10.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.827+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 shutdown_connections 2024-02-10T06:41:10.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:10.828+0000 7f3084fb2700 1 -- 172.21.15.73:0/1533284642 wait complete. 2024-02-10T06:41:10.831 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 14 2024-02-10T06:41:11.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:10 smithi181 ceph-mon[36721]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 409 B/s wr, 2 op/s 2024-02-10T06:41:11.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:11 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1533284642' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:41:11.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:11 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1533284642' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:41:11.897 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-10T06:41:12.166 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:41:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:12.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:12.968 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/1610608955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b40663e0 msgr2=0x7fd0b4101ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/1610608955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b40663e0 0x7fd0b4101ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009ce0 tx=0x7fd0b0004d10 comp rx=0 tx=0).stop 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/1610608955 shutdown_connections 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/1610608955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0b4102430 0x7fd0b4104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/1610608955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b40663e0 0x7fd0b4101ef0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/1610608955 >> 172.21.15.73:0/1610608955 conn(0x7fd0b40f74e0 msgr2=0x7fd0b40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/1610608955 shutdown_connections 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/1610608955 wait complete. 2024-02-10T06:41:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 Processor -- start 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- start start 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0b40663e0 0x7fd0b410fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 0x7fd0b41102d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b4111090 con 0x7fd0b4102430 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.848+0000 7fd0bbb1a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b4111200 con 0x7fd0b40663e0 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 0x7fd0b41102d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 0x7fd0b41102d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53070/0 (socket says 172.21.15.73:53070) 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 -- 172.21.15.73:0/2714383699 learned_addr learned my addr 172.21.15.73:0/2714383699 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b98b6700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0b40663e0 0x7fd0b410fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 -- 172.21.15.73:0/2714383699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0b40663e0 msgr2=0x7fd0b410fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0b40663e0 0x7fd0b410fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 -- 172.21.15.73:0/2714383699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b0009940 con 0x7fd0b4102430 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0b90b5700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 0x7fd0b41102d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4002a50 tx=0x7fd0a4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0a4007cc0 con 0x7fd0b4102430 2024-02-10T06:41:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0a400f040 con 0x7fd0b4102430 2024-02-10T06:41:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0a400d9c0 con 0x7fd0b4102430 2024-02-10T06:41:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b4111480 con 0x7fd0b4102430 2024-02-10T06:41:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.849+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b41119c0 con 0x7fd0b4102430 2024-02-10T06:41:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.850+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b404bed0 con 0x7fd0b4102430 2024-02-10T06:41:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.851+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd0a4009730 con 0x7fd0b4102430 2024-02-10T06:41:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.851+0000 7fd0aa7fc700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd0a00692a0 0x7fd0a006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.851+0000 7fd0b98b6700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd0a00692a0 0x7fd0a006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.851+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd0a4086f80 con 0x7fd0b4102430 2024-02-10T06:41:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.851+0000 7fd0b98b6700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd0a00692a0 0x7fd0a006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009470 tx=0x7fd0b0004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:13.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:13.855+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd0a40531e0 con 0x7fd0b4102430 2024-02-10T06:41:14.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.082+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0b4107c30 con 0x7fd0b4102430 2024-02-10T06:41:14.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.083+0000 7fd0aa7fc700 1 -- 172.21.15.73:0/2714383699 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4694 (secure 0 0 0) 0x7fd0a4056800 con 0x7fd0b4102430 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd0a00692a0 msgr2=0x7fd0a006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd0a00692a0 0x7fd0a006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009470 tx=0x7fd0b0004d10 comp rx=0 tx=0).stop 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 msgr2=0x7fd0b41102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 0x7fd0b41102d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4002a50 tx=0x7fd0a4004600 comp rx=0 tx=0).stop 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 shutdown_connections 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd0a00692a0 0x7fd0a006b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0b40663e0 0x7fd0b410fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 --2- 172.21.15.73:0/2714383699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd0b4102430 0x7fd0b41102d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 >> 172.21.15.73:0/2714383699 conn(0x7fd0b40f74e0 msgr2=0x7fd0b40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:14.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 shutdown_connections 2024-02-10T06:41:14.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:14.086+0000 7fd0bbb1a700 1 -- 172.21.15.73:0/2714383699 wait complete. 2024-02-10T06:41:14.089 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 14 2024-02-10T06:41:14.098 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-02-10T06:41:14.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:14 smithi073 ceph-mon[29063]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 4 op/s 2024-02-10T06:41:14.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:14 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2714383699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:41:14.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:14 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:14.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:14 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:14.870 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-10T06:41:15.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:14 smithi181 ceph-mon[36721]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 4 op/s 2024-02-10T06:41:15.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:14 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2714383699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:41:15.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:14 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:15.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:14 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:17.026 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:16 smithi073 ceph-mon[29063]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 4 op/s 2024-02-10T06:41:17.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:16 smithi181 ceph-mon[36721]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 4 op/s 2024-02-10T06:41:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 -- 172.21.15.73:0/215431436 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fe0e0 msgr2=0x7f5ac00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/215431436 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fe0e0 0x7f5ac00fe550 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4004940 tx=0x7f5ab4009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 -- 172.21.15.73:0/215431436 shutdown_connections 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/215431436 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fe0e0 0x7f5ac00fe550 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/215431436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac00fcef0 0x7f5ac00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 -- 172.21.15.73:0/215431436 >> 172.21.15.73:0/215431436 conn(0x7f5ac00f8620 msgr2=0x7f5ac00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 -- 172.21.15.73:0/215431436 shutdown_connections 2024-02-10T06:41:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 -- 172.21.15.73:0/215431436 wait complete. 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.930+0000 7f5ac60e8700 1 Processor -- start 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5ac60e8700 1 -- start start 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5ac60e8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 0x7f5ac010d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5ac60e8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac00fe0e0 0x7f5ac010deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5ac60e8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac010e4b0 con 0x7f5ac00fcef0 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5ac60e8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac010ed50 con 0x7f5ac00fe0e0 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 0x7f5ac010d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 0x7f5ac010d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53080/0 (socket says 172.21.15.73:53080) 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 -- 172.21.15.73:0/3933528772 learned_addr learned my addr 172.21.15.73:0/3933528772 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abeffd700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac00fe0e0 0x7f5ac010deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 -- 172.21.15.73:0/3933528772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac00fe0e0 msgr2=0x7f5ac010deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:17.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac00fe0e0 0x7f5ac010deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 -- 172.21.15.73:0/3933528772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4005040 con 0x7f5ac00fcef0 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5abf7fe700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 0x7f5ac010d970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0002a80 tx=0x7f5ab0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab0007cc0 con 0x7f5ac00fcef0 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac010f030 con 0x7f5ac00fcef0 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ab000f040 con 0x7f5ac00fcef0 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.931+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab000dad0 con 0x7f5ac00fcef0 2024-02-10T06:41:17.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.932+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac010f570 con 0x7f5ac00fcef0 2024-02-10T06:41:17.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.933+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac004bed0 con 0x7f5ac00fcef0 2024-02-10T06:41:17.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.933+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5ab000dc30 con 0x7f5ac00fcef0 2024-02-10T06:41:17.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.933+0000 7f5aa7fff700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5aa80692a0 0x7f5aa806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:17.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.933+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5ab0086f70 con 0x7f5ac00fcef0 2024-02-10T06:41:17.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.933+0000 7f5abeffd700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5aa80692a0 0x7f5aa806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:17.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.934+0000 7f5abeffd700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5aa80692a0 0x7f5aa806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4004f90 tx=0x7f5ab4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:17.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:17.938+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5ab0056c60 con 0x7f5ac00fcef0 2024-02-10T06:41:18.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.202+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f5ac0105a30 con 0x7f5ac00fcef0 2024-02-10T06:41:18.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.202+0000 7f5aa7fff700 1 -- 172.21.15.73:0/3933528772 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+99 (secure 0 0 0) 0x7f5ab00567f0 con 0x7f5ac00fcef0 2024-02-10T06:41:18.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5aa80692a0 msgr2=0x7f5aa806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:18.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5aa80692a0 0x7f5aa806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4004f90 tx=0x7f5ab4009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:18.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 msgr2=0x7f5ac010d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:18.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 0x7f5ac010d970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0002a80 tx=0x7f5ab0004600 comp rx=0 tx=0).stop 2024-02-10T06:41:18.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 shutdown_connections 2024-02-10T06:41:18.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5aa80692a0 0x7f5aa806b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ac00fcef0 0x7f5ac010d970 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 --2- 172.21.15.73:0/3933528772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac00fe0e0 0x7f5ac010deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 >> 172.21.15.73:0/3933528772 conn(0x7f5ac00f8620 msgr2=0x7f5ac01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:18.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 shutdown_connections 2024-02-10T06:41:18.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.204+0000 7f5ac60e8700 1 -- 172.21.15.73:0/3933528772 wait complete. 2024-02-10T06:41:18.215 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-02-10T06:41:18.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:18 smithi073 ceph-mon[29063]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:41:18.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:18 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3933528772' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-10T06:41:18.840 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-10T06:41:18.852 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-10T06:41:18.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 -- 172.21.15.73:0/1336851797 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fd540 msgr2=0x7fd29c0fd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:18.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 --2- 172.21.15.73:0/1336851797 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fd540 0x7fd29c0fd9b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd2900048d0 tx=0x7fd290009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 -- 172.21.15.73:0/1336851797 shutdown_connections 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 --2- 172.21.15.73:0/1336851797 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fd540 0x7fd29c0fd9b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 --2- 172.21.15.73:0/1336851797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fc270 0x7fd29c0fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 -- 172.21.15.73:0/1336851797 >> 172.21.15.73:0/1336851797 conn(0x7fd29c0f7a30 msgr2=0x7fd29c0f9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 -- 172.21.15.73:0/1336851797 shutdown_connections 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.981+0000 7fd2a139d640 1 -- 172.21.15.73:0/1336851797 wait complete. 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd2a139d640 1 Processor -- start 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd2a139d640 1 -- start start 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd2a139d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 0x7fd29c10cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd2a139d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fd540 0x7fd29c10d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd2a139d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c10d7c0 con 0x7fd29c0fc270 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd2a139d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c112180 con 0x7fd29c0fd540 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd29affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 0x7fd29c10cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:18.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd29affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 0x7fd29c10cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51706/0 (socket says 172.21.15.73:51706) 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd29affd640 1 -- 172.21.15.73:0/2192940602 learned_addr learned my addr 172.21.15.73:0/2192940602 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.982+0000 7fd29a7fc640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fd540 0x7fd29c10d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd29affd640 1 -- 172.21.15.73:0/2192940602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fd540 msgr2=0x7fd29c10d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd29affd640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fd540 0x7fd29c10d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd29affd640 1 -- 172.21.15.73:0/2192940602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290005040 con 0x7fd29c0fc270 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd29a7fc640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fd540 0x7fd29c10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd29affd640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 0x7fd29c10cba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00b790 tx=0x7fd28c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28c00c700 con 0x7fd29c0fc270 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29c112380 con 0x7fd29c0fc270 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd28c00cd30 con 0x7fd29c0fc270 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28c013370 con 0x7fd29c0fc270 2024-02-10T06:41:18.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.983+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c1128a0 con 0x7fd29c0fc270 2024-02-10T06:41:18.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.984+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c104e40 con 0x7fd29c0fc270 2024-02-10T06:41:18.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.984+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd28c012070 con 0x7fd29c0fc270 2024-02-10T06:41:18.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.985+0000 7fd2837fe640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd284069210 0x7fd28406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:18.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.985+0000 7fd29a7fc640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd284069210 0x7fd28406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:18.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.985+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd28c086eb0 con 0x7fd29c0fc270 2024-02-10T06:41:18.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.985+0000 7fd29a7fc640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd284069210 0x7fd28406b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd29c10e0f0 tx=0x7fd290009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:18.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:18.989+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd28c053110 con 0x7fd29c0fc270 2024-02-10T06:41:19.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:18 smithi181 ceph-mon[36721]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:41:19.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:18 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3933528772' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-10T06:41:19.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.155+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd29c112b80 con 0x7fd29c0fc270 2024-02-10T06:41:19.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.156+0000 7fd2837fe640 1 -- 172.21.15.73:0/2192940602 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14385 (secure 0 0 0) 0x7fd28c056730 con 0x7fd29c0fc270 2024-02-10T06:41:19.157 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:19.157 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":47,"fsid":"3c2b87a2-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:33:49.428986+0000","modified":"2024-02-10T06:41:05.648697+0000","last_up_change":"2024-02-10T06:39:28.526237+0000","last_in_change":"2024-02-10T06:39:10.116034+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:38:17.748559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-10T06:40:49.775412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-10T06:40:50.065514+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8e333d6f-68ed-41e7-a417-bed0ad6e2ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6803","nonce":1003470321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6805","nonce":1003470321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6809","nonce":1003470321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1003470321},{"type":"v1","addr":"172.21.15.73:6807","nonce":1003470321}]},"public_addr":"172.21.15.73:6803/1003470321","cluster_addr":"172.21.15.73:6805/1003470321","heartbeat_back_addr":"172.21.15.73:6809/1003470321","heartbeat_front_addr":"172.21.15.73:6807/1003470321","state":["exists","up"]},{"osd":1,"uuid":"dedbb82b-7b4d-47eb-8621-2a35fa20a05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6811","nonce":2186552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6813","nonce":2186552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6817","nonce":2186552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":2186552820},{"type":"v1","addr":"172.21.15.73:6815","nonce":2186552820}]},"public_addr":"172.21.15.73:6811/2186552820","cluster_addr":"172.21.15.73:6813/2186552820","heartbeat_back_addr":"172.21.15.73:6817/2186552820","heartbeat_front_addr":"172.21.15.73:6815/2186552820","state":["exists","up"]},{"osd":2,"uuid":"6e7d61ae-a65b-43db-ac2d-aec724834f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6819","nonce":4245951128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6821","nonce":4245951128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6825","nonce":4245951128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":4245951128},{"type":"v1","addr":"172.21.15.73:6823","nonce":4245951128}]},"public_addr":"172.21.15.73:6819/4245951128","cluster_addr":"172.21.15.73:6821/4245951128","heartbeat_back_addr":"172.21.15.73:6825/4245951128","heartbeat_front_addr":"172.21.15.73:6823/4245951128","state":["exists","up"]},{"osd":3,"uuid":"2f17b271-99d5-4b38-89fb-175414c59351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6801","nonce":156965919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6803","nonce":156965919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6807","nonce":156965919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":156965919},{"type":"v1","addr":"172.21.15.181:6805","nonce":156965919}]},"public_addr":"172.21.15.181:6801/156965919","cluster_addr":"172.21.15.181:6803/156965919","heartbeat_back_addr":"172.21.15.181:6807/156965919","heartbeat_front_addr":"172.21.15.181:6805/156965919","state":["exists","up"]},{"osd":4,"uuid":"28fd87c6-5f23-44af-87a6-d3cf81095f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6809","nonce":849727482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6811","nonce":849727482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6815","nonce":849727482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":849727482},{"type":"v1","addr":"172.21.15.181:6813","nonce":849727482}]},"public_addr":"172.21.15.181:6809/849727482","cluster_addr":"172.21.15.181:6811/849727482","heartbeat_back_addr":"172.21.15.181:6815/849727482","heartbeat_front_addr":"172.21.15.181:6813/849727482","state":["exists","up"]},{"osd":5,"uuid":"9bd38e72-1e2b-41a3-974c-c5d431919187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6817","nonce":1157810956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6819","nonce":1157810956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6823","nonce":1157810956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":1157810956},{"type":"v1","addr":"172.21.15.181:6821","nonce":1157810956}]},"public_addr":"172.21.15.181:6817/1157810956","cluster_addr":"172.21.15.181:6819/1157810956","heartbeat_back_addr":"172.21.15.181:6823/1157810956","heartbeat_front_addr":"172.21.15.181:6821/1157810956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:14.313049+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:37:44.358563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:14.291417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:38:37.782333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:02.393199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:26.947433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/194898351":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/1643478797":"2024-02-11T06:34:48.820922+0000","172.21.15.73:0/810500280":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6800/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6800/194898351":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/2932627761":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/1695656680":"2024-02-11T06:34:17.719875+0000","172.21.15.73:6827/3433504765":"2024-02-11T06:41:05.648672+0000","172.21.15.73:0/3298620834":"2024-02-11T06:34:17.719875+0000","172.21.15.73:0/3119085620":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6801/1157895579":"2024-02-11T06:34:48.820922+0000","172.21.15.73:6801/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/1749918597":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6800/2308737934":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/4190786023":"2024-02-11T06:35:57.420666+0000","172.21.15.73:0/3920786161":"2024-02-11T06:35:57.420666+0000","172.21.15.73:6826/3433504765":"2024-02-11T06:41:05.648672+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd284069210 msgr2=0x7fd28406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd284069210 0x7fd28406b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd29c10e0f0 tx=0x7fd290009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 msgr2=0x7fd29c10cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 0x7fd29c10cba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00b790 tx=0x7fd28c0091d0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 shutdown_connections 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fd284069210 0x7fd28406b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd29c0fd540 0x7fd29c10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 --2- 172.21.15.73:0/2192940602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd29c0fc270 0x7fd29c10cba0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 >> 172.21.15.73:0/2192940602 conn(0x7fd29c0f7a30 msgr2=0x7fd29c100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:19.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 shutdown_connections 2024-02-10T06:41:19.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.158+0000 7fd2a139d640 1 -- 172.21.15.73:0/2192940602 wait complete. 2024-02-10T06:41:19.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T06:38:17.748559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-10T06:40:49.775412+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-10T06:40:50.065514+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-10T06:41:19.170 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-10T06:41:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 -- 172.21.15.73:0/4076224739 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fb1d0 msgr2=0x7feb280fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 --2- 172.21.15.73:0/4076224739 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fb1d0 0x7feb280fb5e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7feb180048d0 tx=0x7feb18009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 -- 172.21.15.73:0/4076224739 shutdown_connections 2024-02-10T06:41:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 --2- 172.21.15.73:0/4076224739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fc400 0x7feb280fc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 --2- 172.21.15.73:0/4076224739 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fb1d0 0x7feb280fb5e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 -- 172.21.15.73:0/4076224739 >> 172.21.15.73:0/4076224739 conn(0x7feb280f68d0 msgr2=0x7feb280f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 -- 172.21.15.73:0/4076224739 shutdown_connections 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.276+0000 7feb2d42e640 1 -- 172.21.15.73:0/4076224739 wait complete. 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb2d42e640 1 Processor -- start 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb2d42e640 1 -- start start 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb2d42e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fb1d0 0x7feb2810cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb2d42e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 0x7feb2810d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb2d42e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2810d860 con 0x7feb280fc400 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb2d42e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb281121e0 con 0x7feb280fb1d0 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 0x7feb2810d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 0x7feb2810d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51726/0 (socket says 172.21.15.73:51726) 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 -- 172.21.15.73:0/1970059402 learned_addr learned my addr 172.21.15.73:0/1970059402 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb26ffd640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fb1d0 0x7feb2810cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 -- 172.21.15.73:0/1970059402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fb1d0 msgr2=0x7feb2810cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fb1d0 0x7feb2810cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 -- 172.21.15.73:0/1970059402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18005040 con 0x7feb280fc400 2024-02-10T06:41:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb26ffd640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fb1d0 0x7feb2810cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.277+0000 7feb1ffff640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 0x7feb2810d180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feb1000b790 tx=0x7feb100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.278+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb1000c700 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.278+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb1000cd30 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.278+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb281123e0 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.278+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb10013370 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.278+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb28112900 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.279+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7feb10012070 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.279+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb28103da0 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.279+0000 7feb1f7fe640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7feb14069220 0x7feb1406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.279+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7feb10086e70 con 0x7feb280fc400 2024-02-10T06:41:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.279+0000 7feb26ffd640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7feb14069220 0x7feb1406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.280+0000 7feb26ffd640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7feb14069220 0x7feb1406b6d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb18003080 tx=0x7feb18009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.284+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7feb100530d0 con 0x7feb280fc400 2024-02-10T06:41:19.484 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.483+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb28103f30 con 0x7feb280fc400 2024-02-10T06:41:19.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.484+0000 7feb1f7fe640 1 -- 172.21.15.73:0/1970059402 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb28103f30 con 0x7feb280fc400 2024-02-10T06:41:19.485 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-02-10T06:41:19.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7feb14069220 msgr2=0x7feb1406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7feb14069220 0x7feb1406b6d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb18003080 tx=0x7feb18009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 msgr2=0x7feb2810d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 0x7feb2810d180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feb1000b790 tx=0x7feb100091d0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 shutdown_connections 2024-02-10T06:41:19.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7feb14069220 0x7feb1406b6d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb280fc400 0x7feb2810d180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 --2- 172.21.15.73:0/1970059402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb280fb1d0 0x7feb2810cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 >> 172.21.15.73:0/1970059402 conn(0x7feb280f68d0 msgr2=0x7feb28064480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:19.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 shutdown_connections 2024-02-10T06:41:19.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.486+0000 7feb2d42e640 1 -- 172.21.15.73:0/1970059402 wait complete. 2024-02-10T06:41:19.498 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-10T06:41:19.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.597+0000 7f5c6915d640 1 -- 172.21.15.73:0/4040795934 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fc450 msgr2=0x7f5c640fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.597+0000 7f5c6915d640 1 --2- 172.21.15.73:0/4040795934 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fc450 0x7f5c640fc8c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0048d0 tx=0x7f5c4c009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.597+0000 7f5c6915d640 1 -- 172.21.15.73:0/4040795934 shutdown_connections 2024-02-10T06:41:19.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.597+0000 7f5c6915d640 1 --2- 172.21.15.73:0/4040795934 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fc450 0x7f5c640fc8c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.597+0000 7f5c6915d640 1 --2- 172.21.15.73:0/4040795934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c640fb180 0x7f5c640fb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.597+0000 7f5c6915d640 1 -- 172.21.15.73:0/4040795934 >> 172.21.15.73:0/4040795934 conn(0x7f5c640f6960 msgr2=0x7f5c640f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 -- 172.21.15.73:0/4040795934 shutdown_connections 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 -- 172.21.15.73:0/4040795934 wait complete. 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 Processor -- start 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 -- start start 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 0x7f5c6410ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c640fc450 0x7f5c6410bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6410c5e0 con 0x7f5c640fb180 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c6915d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c64111030 con 0x7f5c640fc450 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c62d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 0x7f5c6410ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c62d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 0x7f5c6410ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51742/0 (socket says 172.21.15.73:51742) 2024-02-10T06:41:19.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.598+0000 7f5c62d9d640 1 -- 172.21.15.73:0/3003460391 learned_addr learned my addr 172.21.15.73:0/3003460391 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c6259c640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c640fc450 0x7f5c6410bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c62d9d640 1 -- 172.21.15.73:0/3003460391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c640fc450 msgr2=0x7f5c6410bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c62d9d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c640fc450 0x7f5c6410bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c62d9d640 1 -- 172.21.15.73:0/3003460391 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c005040 con 0x7f5c640fb180 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c62d9d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 0x7f5c6410ba50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00cb10 tx=0x7f5c5c007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5c004200 con 0x7f5c640fb180 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c641111d0 con 0x7f5c640fb180 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c5c00acf0 con 0x7f5c640fb180 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5c003e40 con 0x7f5c640fb180 2024-02-10T06:41:19.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.599+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c641116c0 con 0x7f5c640fb180 2024-02-10T06:41:19.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.600+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c64103cf0 con 0x7f5c640fb180 2024-02-10T06:41:19.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.600+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5c5c00a820 con 0x7f5c640fb180 2024-02-10T06:41:19.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.601+0000 7f5c5b7fe640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5c500691a0 0x7f5c5006b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.601+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5c5c087900 con 0x7f5c640fb180 2024-02-10T06:41:19.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.601+0000 7f5c6259c640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5c500691a0 0x7f5c5006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.601+0000 7f5c6259c640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5c500691a0 0x7f5c5006b650 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c003080 tx=0x7f5c4c009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:19.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.605+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5c5c053c20 con 0x7f5c640fb180 2024-02-10T06:41:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.784+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f5c64103e80 con 0x7f5c640fb180 2024-02-10T06:41:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.785+0000 7f5c5b7fe640 1 -- 172.21.15.73:0/3003460391 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5c64103e80 con 0x7f5c640fb180 2024-02-10T06:41:19.786 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5c500691a0 msgr2=0x7f5c5006b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5c500691a0 0x7f5c5006b650 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c003080 tx=0x7f5c4c009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 msgr2=0x7f5c6410ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 0x7f5c6410ba50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00cb10 tx=0x7f5c5c007ee0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 shutdown_connections 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f5c500691a0 0x7f5c5006b650 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c640fc450 0x7f5c6410bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 --2- 172.21.15.73:0/3003460391 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5c640fb180 0x7f5c6410ba50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 >> 172.21.15.73:0/3003460391 conn(0x7f5c640f6960 msgr2=0x7f5c640ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 shutdown_connections 2024-02-10T06:41:19.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.787+0000 7f5c6915d640 1 -- 172.21.15.73:0/3003460391 wait complete. 2024-02-10T06:41:19.797 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:19 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2192940602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:41:19.797 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:19 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1970059402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:41:19.798 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 -- 172.21.15.73:0/1067466871 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 msgr2=0x7fb58c0fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 --2- 172.21.15.73:0/1067466871 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c0fc6d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009c50 tx=0x7fb57c004d10 comp rx=0 tx=0).stop 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 -- 172.21.15.73:0/1067466871 shutdown_connections 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 --2- 172.21.15.73:0/1067466871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb58c0fd590 0x7fb58c0fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 --2- 172.21.15.73:0/1067466871 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c0fc6d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 -- 172.21.15.73:0/1067466871 >> 172.21.15.73:0/1067466871 conn(0x7fb58c0f7a60 msgr2=0x7fb58c0f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.897+0000 7fb590f1d640 1 -- 172.21.15.73:0/1067466871 shutdown_connections 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 -- 172.21.15.73:0/1067466871 wait complete. 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 Processor -- start 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 -- start start 2024-02-10T06:41:19.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c10cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb58c0fd590 0x7fb58c10d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c10d6e0 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb590f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c112130 con 0x7fb58c0fd590 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb58a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c10cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb58a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c10cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51756/0 (socket says 172.21.15.73:51756) 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb58a59c640 1 -- 172.21.15.73:0/893362777 learned_addr learned my addr 172.21.15.73:0/893362777 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb58a59c640 1 -- 172.21.15.73:0/893362777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb58c0fd590 msgr2=0x7fb58c10d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb58a59c640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb58c0fd590 0x7fb58c10d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.898+0000 7fb58a59c640 1 -- 172.21.15.73:0/893362777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0098b0 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.899+0000 7fb58a59c640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c10cb50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb57c004750 tx=0x7fb57c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.899+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb57c03a070 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.899+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58c112270 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.899+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb57c009720 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.899+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb57c03ea90 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.899+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58c1127e0 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.900+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58c104e30 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.900+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb57c032070 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.900+0000 7fb57affd640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb574069210 0x7fb57406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:19.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.901+0000 7fb589d9b640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb574069210 0x7fb57406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:19.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.901+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb57c0a9690 con 0x7fb58c0fc2c0 2024-02-10T06:41:19.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.901+0000 7fb589d9b640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb574069210 0x7fb57406b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb58c10e0a0 tx=0x7fb580009c50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:19.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:19.905+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb57c0758f0 con 0x7fb58c0fc2c0 2024-02-10T06:41:20.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.084+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb58c112f20 con 0x7fb58c0fc2c0 2024-02-10T06:41:20.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.085+0000 7fb57affd640 1 -- 172.21.15.73:0/893362777 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb57c078f10 con 0x7fb58c0fc2c0 2024-02-10T06:41:20.085 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-02-10T06:41:20.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.086+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb574069210 msgr2=0x7fb57406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.086+0000 7fb590f1d640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb574069210 0x7fb57406b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb58c10e0a0 tx=0x7fb580009c50 comp rx=0 tx=0).stop 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 msgr2=0x7fb58c10cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c10cb50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb57c004750 tx=0x7fb57c004d10 comp rx=0 tx=0).stop 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 shutdown_connections 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb574069210 0x7fb57406b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb58c0fd590 0x7fb58c10d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 --2- 172.21.15.73:0/893362777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb58c0fc2c0 0x7fb58c10cb50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 >> 172.21.15.73:0/893362777 conn(0x7fb58c0f7a60 msgr2=0x7fb58c1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 shutdown_connections 2024-02-10T06:41:20.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.087+0000 7fb590f1d640 1 -- 172.21.15.73:0/893362777 wait complete. 2024-02-10T06:41:20.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:19 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2192940602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:41:20.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:19 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1970059402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:41:20.098 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-10T06:41:20.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 -- 172.21.15.73:0/3424232388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fc2a0 msgr2=0x7f76f00fc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:20.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 --2- 172.21.15.73:0/3424232388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fc2a0 0x7f76f00fc6b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f76e40048d0 tx=0x7f76e4009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 -- 172.21.15.73:0/3424232388 shutdown_connections 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 --2- 172.21.15.73:0/3424232388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fd4d0 0x7f76f00fd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 --2- 172.21.15.73:0/3424232388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fc2a0 0x7f76f00fc6b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 -- 172.21.15.73:0/3424232388 >> 172.21.15.73:0/3424232388 conn(0x7f76f00f79c0 msgr2=0x7f76f00f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 -- 172.21.15.73:0/3424232388 shutdown_connections 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 -- 172.21.15.73:0/3424232388 wait complete. 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 Processor -- start 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.195+0000 7f76f499d640 1 -- start start 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76f499d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fc2a0 0x7f76f010cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76f499d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 0x7f76f010d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76f499d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f010d6e0 con 0x7f76f00fd4d0 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76f499d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0112130 con 0x7f76f00fc2a0 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 0x7f76f010d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 0x7f76f010d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51770/0 (socket says 172.21.15.73:51770) 2024-02-10T06:41:20.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 -- 172.21.15.73:0/3828934880 learned_addr learned my addr 172.21.15.73:0/3828934880 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76ee59c640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fc2a0 0x7f76f010cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 -- 172.21.15.73:0/3828934880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fc2a0 msgr2=0x7f76f010cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fc2a0 0x7f76f010cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 -- 172.21.15.73:0/3828934880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4005040 con 0x7f76f00fd4d0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76ee59c640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fc2a0 0x7f76f010cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76edd9b640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 0x7f76f010d090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c830 tx=0x7f76e0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e000a850 con 0x7f76f00fd4d0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76e000ae80 con 0x7f76f00fd4d0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f01122d0 con 0x7f76f00fd4d0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e0012ea0 con 0x7f76f00fd4d0 2024-02-10T06:41:20.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.196+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f01127c0 con 0x7f76f00fd4d0 2024-02-10T06:41:20.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.197+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0104df0 con 0x7f76f00fd4d0 2024-02-10T06:41:20.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.197+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f76e0012400 con 0x7f76f00fd4d0 2024-02-10T06:41:20.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.198+0000 7f76deffd640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f76d8069210 0x7f76d806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:20.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.198+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f76e0087450 con 0x7f76f00fd4d0 2024-02-10T06:41:20.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.198+0000 7f76ee59c640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f76d8069210 0x7f76d806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:20.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.198+0000 7f76ee59c640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f76d8069210 0x7f76d806b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f76e4000c00 tx=0x7f76e4009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:20.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.202+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f76e0053730 con 0x7f76f00fd4d0 2024-02-10T06:41:20.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.402+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76f0112a40 con 0x7f76f00fd4d0 2024-02-10T06:41:20.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.403+0000 7f76deffd640 1 -- 172.21.15.73:0/3828934880 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4694 (secure 0 0 0) 0x7f76e00126b0 con 0x7f76f00fd4d0 2024-02-10T06:41:20.404 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:20.404 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14476,"name":"cephfs.smithi073.ykxzap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.73:6829/3906379226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":3906379226},{"type":"v1","addr":"172.21.15.73:6829","nonce":3906379226}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24249,"name":"cephfs.smithi181.otunfp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.181:6827/1480345027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":1480345027},{"type":"v1","addr":"172.21.15.181:6827","nonce":1480345027}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24257,"name":"cephfs.smithi073.nuvejb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6827/2661433887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":2661433887},{"type":"v1","addr":"172.21.15.73:6827","nonce":2661433887}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-10T06:40:50.835658+0000","modified":"2024-02-10T06:41:08.652904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24237},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24237":{"gid":24237,"name":"cephfs.smithi181.cuebex","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6825/3933807667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3933807667},{"type":"v1","addr":"172.21.15.181:6825","nonce":3933807667}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-10T06:41:20.404 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 14 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f76d8069210 msgr2=0x7f76d806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f76d8069210 0x7f76d806b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f76e4000c00 tx=0x7f76e4009da0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 msgr2=0x7f76f010d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 0x7f76f010d090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c830 tx=0x7f76e0004430 comp rx=0 tx=0).stop 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 shutdown_connections 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f76d8069210 0x7f76d806b6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76f00fd4d0 0x7f76f010d090 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 --2- 172.21.15.73:0/3828934880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76f00fc2a0 0x7f76f010cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:20.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 >> 172.21.15.73:0/3828934880 conn(0x7f76f00f79c0 msgr2=0x7f76f0100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:20.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 shutdown_connections 2024-02-10T06:41:20.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:20.405+0000 7f76f499d640 1 -- 172.21.15.73:0/3828934880 wait complete. 2024-02-10T06:41:20.416 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 1, 'flags': 18} 2024-02-10T06:41:20.416 INFO:teuthology.run_tasks:Running task kclient... 2024-02-10T06:41:20.441 INFO:tasks.kclient:Mounting kernel clients... 2024-02-10T06:41:20.441 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-10T06:41:20.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:20.441 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-02-10T06:41:20.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:20.456 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link delete ceph-brx 2024-02-10T06:41:20.526 INFO:teuthology.orchestra.run.smithi073.stderr:Cannot find device "ceph-brx" 2024-02-10T06:41:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:41:20.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:20.528 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-02-10T06:41:20.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:20.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link delete ceph-brx 2024-02-10T06:41:20.624 INFO:teuthology.orchestra.run.smithi181.stderr:Cannot find device "ceph-brx" 2024-02-10T06:41:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:41:20.626 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-10T06:41:20.626 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-10T06:41:20.626 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:20.626 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-10T06:41:20.901 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:21.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:20 smithi073 ceph-mon[29063]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:41:21.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:20 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3003460391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-10T06:41:21.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:20 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/893362777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-10T06:41:21.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:20 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3828934880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:41:21.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:20 smithi181 ceph-mon[36721]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:41:21.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:20 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3003460391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-10T06:41:21.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:20 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/893362777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-10T06:41:21.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:20 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3828934880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:41:21.840 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:21 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:21.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:21 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:21.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:21 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:21.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:21 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:21.841 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:21 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:21 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:21 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:21 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:22.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:21 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:22.122 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:21 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:22.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.553+0000 7f7489f0f700 1 -- 172.21.15.73:0/1248863181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 msgr2=0x7f74840706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:22.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.553+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1248863181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f74840706a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7474004970 tx=0x7f7474009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:22.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.553+0000 7f7489f0f700 1 -- 172.21.15.73:0/1248863181 shutdown_connections 2024-02-10T06:41:22.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.553+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1248863181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 0x7f74841092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1248863181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f74840706a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 -- 172.21.15.73:0/1248863181 >> 172.21.15.73:0/1248863181 conn(0x7f74840f8ae0 msgr2=0x7f74840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 -- 172.21.15.73:0/1248863181 shutdown_connections 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 -- 172.21.15.73:0/1248863181 wait complete. 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 Processor -- start 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 -- start start 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f7484115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 0x7f74841161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841168c0 con 0x7f74840702d0 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f7489f0f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748410fb40 con 0x7f7484070be0 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.554+0000 7f74837fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f7484115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f74837fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f7484115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51784/0 (socket says 172.21.15.73:51784) 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f74837fe700 1 -- 172.21.15.73:0/1759485422 learned_addr learned my addr 172.21.15.73:0/1759485422 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:22.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f7482ffd700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 0x7f74841161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f74837fe700 1 -- 172.21.15.73:0/1759485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 msgr2=0x7f74841161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f74837fe700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 0x7f74841161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f74837fe700 1 -- 172.21.15.73:0/1759485422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474005040 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f7482ffd700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 0x7f74841161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f74837fe700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f7484115ca0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7474000c00 tx=0x7f7474009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747400e040 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7474004290 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748410fd60 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747401c630 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.555+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74841102a0 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.556+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7484106980 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.557+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7474013980 con 0x7f74840702d0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.557+0000 7f746bfff700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f746c0692a0 0x7f746c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.557+0000 7f7482ffd700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f746c0692a0 0x7f746c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:22.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.557+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f74740884d0 con 0x7f74840702d0 2024-02-10T06:41:22.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.557+0000 7f7482ffd700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f746c0692a0 0x7f746c06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7478006fd0 tx=0x7f7478008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:22.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.561+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7474058240 con 0x7f74840702d0 2024-02-10T06:41:22.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.790+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7484111280 con 0x7f74840702d0 2024-02-10T06:41:22.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.790+0000 7f746bfff700 1 -- 172.21.15.73:0/1759485422 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f7474057dd0 con 0x7f74840702d0 2024-02-10T06:41:22.791 INFO:teuthology.orchestra.run.smithi073.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-10T06:41:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f746c0692a0 msgr2=0x7f746c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f746c0692a0 0x7f746c06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7478006fd0 tx=0x7f7478008040 comp rx=0 tx=0).stop 2024-02-10T06:41:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 msgr2=0x7f7484115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f7484115ca0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7474000c00 tx=0x7f7474009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 shutdown_connections 2024-02-10T06:41:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f746c0692a0 0x7f746c06b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74840702d0 0x7f7484115ca0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 --2- 172.21.15.73:0/1759485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7484070be0 0x7f74841161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 >> 172.21.15.73:0/1759485422 conn(0x7f74840f8ae0 msgr2=0x7f7484103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.792+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 shutdown_connections 2024-02-10T06:41:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:22.793+0000 7f7489f0f700 1 -- 172.21.15.73:0/1759485422 wait complete. 2024-02-10T06:41:23.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:22 smithi181 ceph-mon[36721]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:41:23.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:22 smithi073 ceph-mon[29063]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:41:23.484 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-10T06:41:23.484 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-10T06:41:23.484 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi073.front.sepia.ceph.com 2024-02-10T06:41:23.484 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-10T06:41:23.485 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:23.485 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-10T06:41:23.485 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-10T06:41:23.485 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-10T06:41:23.485 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-10T06:41:23.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:23.485 DEBUG:teuthology.orchestra.run.smithi073:> ip addr 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: inet 127.0.0.1/8 scope host lo 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 ::1/128 scope host 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:ea brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f0 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:eb brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f1 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:41:23.502 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7c brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run.smithi073.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7d brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run.smithi073.stdout: inet 172.21.15.73/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft 42305sec preferred_lft 42305sec 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 fe80::ec4:7aff:fe8f:ce7d/64 scope link 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:41:23.503 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-10T06:41:23.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add name ceph-brx type bridge 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr flush dev ceph-brx 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set ceph-brx up 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-10T06:41:23.504 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-02-10T06:41:23.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:23.697 DEBUG:teuthology.orchestra.run.smithi073:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-10T06:41:23.723 INFO:teuthology.orchestra.run.smithi073.stdout:1 2024-02-10T06:41:23.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:23.725 DEBUG:teuthology.orchestra.run.smithi073:> route 2024-02-10T06:41:23.791 INFO:teuthology.orchestra.run.smithi073.stdout:Kernel IP routing table 2024-02-10T06:41:23.791 INFO:teuthology.orchestra.run.smithi073.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-10T06:41:23.791 INFO:teuthology.orchestra.run.smithi073.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-10T06:41:23.792 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-10T06:41:23.792 INFO:teuthology.orchestra.run.smithi073.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-10T06:41:23.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:23.792 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-02-10T06:41:23.792 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-02-10T06:41:23.792 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-10T06:41:23.792 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-10T06:41:23.792 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-10T06:41:23.793 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-02-10T06:41:23.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:23 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1759485422' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:41:24.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.069 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-02-10T06:41:24.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.084 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list-id 2024-02-10T06:41:24.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:23 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1759485422' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:41:24.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.139 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-02-10T06:41:24.139 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-02-10T06:41:24.139 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-10T06:41:24.139 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-10T06:41:24.140 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-02-10T06:41:24.222 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-10T06:41:24.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.222 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-02-10T06:41:24.222 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-02-10T06:41:24.223 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-10T06:41:24.223 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-10T06:41:24.223 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-10T06:41:24.223 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-10T06:41:24.223 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-10T06:41:24.223 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-02-10T06:41:24.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.365 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-02-10T06:41:24.365 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-02-10T06:41:24.365 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set brx.0 up 2024-02-10T06:41:24.365 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set dev brx.0 master ceph-brx 2024-02-10T06:41:24.365 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-02-10T06:41:24.406 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:41:24.407 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:24.461 INFO:teuthology.orchestra.run.smithi073.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-10T06:41:24.461 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:41:24.461 DEBUG:teuthology.orchestra.run.smithi073:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:24.516 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-10T06:41:24.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.516 DEBUG:teuthology.orchestra.run.smithi073:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:41:24.601 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:41:24.601 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-10T06:41:24.601 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-02-10T06:41:24.707 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:41:24.707 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-10T06:41:24.707 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-02-10T06:41:24.707 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.73:3300,172.21.15.181:3300" 2024-02-10T06:41:24.707 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.73:3300,172.21.15.181:3300" 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.73:6789,172.21.15.181:6789" 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.73:6789,172.21.15.181:6789:/ 2024-02-10T06:41:24.713 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=3c2b87a2-c7de-11ee-95b8-87774f69a715" will pass to kernel 2024-02-10T06:41:24.714 INFO:teuthology.orchestra.run.smithi073.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-10T06:41:24.714 INFO:teuthology.orchestra.run.smithi073.stdout: You just mounted a file system that supports labels which does not 2024-02-10T06:41:24.714 INFO:teuthology.orchestra.run.smithi073.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-10T06:41:24.714 INFO:teuthology.orchestra.run.smithi073.stdout: applications will generate AVC messages and not be allowed access to 2024-02-10T06:41:24.714 INFO:teuthology.orchestra.run.smithi073.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-10T06:41:24.715 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-10T06:41:24.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.716 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:24.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.743 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:41:24.809 INFO:teuthology.orchestra.run.smithi073.stdout:client24273 2024-02-10T06:41:24.809 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:41:24.809 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:41:24.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.866 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:41:24.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:24 smithi073 ceph-mon[29063]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:41:24.896 INFO:teuthology.orchestra.run.smithi073.stdout:client24273 2024-02-10T06:41:24.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.896 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:41:24.962 INFO:teuthology.orchestra.run.smithi073.stdout:client24273 2024-02-10T06:41:24.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:24.963 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715.client24273/status') 2024-02-10T06:41:25.035 INFO:teuthology.orchestra.run.smithi073.stdout:instance: client.24273 (3)172.21.15.73:0/952131128 2024-02-10T06:41:25.035 INFO:teuthology.orchestra.run.smithi073.stdout:blocklisted: false 2024-02-10T06:41:25.036 INFO:teuthology.orchestra.run.smithi073.stderr:0+1 records in 2024-02-10T06:41:25.036 INFO:teuthology.orchestra.run.smithi073.stderr:0+1 records out 2024-02-10T06:41:25.036 INFO:teuthology.orchestra.run.smithi073.stderr:70 bytes copied, 2.0821e-05 s, 3.4 MB/s 2024-02-10T06:41:25.037 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-10T06:41:25.037 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-10T06:41:25.037 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:25.037 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-10T06:41:25.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:24 smithi181 ceph-mon[36721]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:41:25.334 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 -- 172.21.15.73:0/3266657728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 msgr2=0x7fad380fb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 --2- 172.21.15.73:0/3266657728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fb9d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad34004940 tx=0x7fad34009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 -- 172.21.15.73:0/3266657728 shutdown_connections 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 --2- 172.21.15.73:0/3266657728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fb9d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 --2- 172.21.15.73:0/3266657728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad38102fe0 0x7fad381033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 -- 172.21.15.73:0/3266657728 >> 172.21.15.73:0/3266657728 conn(0x7fad38071500 msgr2=0x7fad38071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 -- 172.21.15.73:0/3266657728 shutdown_connections 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.001+0000 7fad40287700 1 -- 172.21.15.73:0/3266657728 wait complete. 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad40287700 1 Processor -- start 2024-02-10T06:41:27.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad40287700 1 -- start start 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad40287700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad40287700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad38102fe0 0x7fad380fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad40287700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380fd2a0 con 0x7fad380fb560 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad40287700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380fd410 con 0x7fad38102fe0 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51814/0 (socket says 172.21.15.73:51814) 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 -- 172.21.15.73:0/2847917113 learned_addr learned my addr 172.21.15.73:0/2847917113 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3d822700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad38102fe0 0x7fad380fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 -- 172.21.15.73:0/2847917113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad38102fe0 msgr2=0x7fad380fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad38102fe0 0x7fad380fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 -- 172.21.15.73:0/2847917113 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad34005040 con 0x7fad380fb560 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.002+0000 7fad3e023700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fc740 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fad2800b790 tx=0x7fad28009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.003+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2800f6f0 con 0x7fad380fb560 2024-02-10T06:41:27.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.003+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad2800fd20 con 0x7fad380fb560 2024-02-10T06:41:27.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.003+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad38119d40 con 0x7fad380fb560 2024-02-10T06:41:27.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.003+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad280103a0 con 0x7fad380fb560 2024-02-10T06:41:27.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.003+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3811a220 con 0x7fad380fb560 2024-02-10T06:41:27.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.004+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad38100290 con 0x7fad380fb560 2024-02-10T06:41:27.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.004+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fad28011400 con 0x7fad380fb560 2024-02-10T06:41:27.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.004+0000 7fad2effd700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fad240692a0 0x7fad2406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:27.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.004+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fad28087020 con 0x7fad380fb560 2024-02-10T06:41:27.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.005+0000 7fad3d822700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fad240692a0 0x7fad2406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:27.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.005+0000 7fad3d822700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fad240692a0 0x7fad2406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad3400b3b0 tx=0x7fad34009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:27.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.009+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fad28053280 con 0x7fad380fb560 2024-02-10T06:41:27.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:26 smithi181 ceph-mon[36721]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-10T06:41:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:26 smithi073 ceph-mon[29063]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-10T06:41:27.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.233+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fad3804bf30 con 0x7fad380fb560 2024-02-10T06:41:27.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.234+0000 7fad2effd700 1 -- 172.21.15.73:0/2847917113 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7fad280568a0 con 0x7fad380fb560 2024-02-10T06:41:27.234 INFO:teuthology.orchestra.run.smithi073.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-10T06:41:27.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.235+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fad240692a0 msgr2=0x7fad2406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:27.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.235+0000 7fad40287700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fad240692a0 0x7fad2406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad3400b3b0 tx=0x7fad34009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.235+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 msgr2=0x7fad380fc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.235+0000 7fad40287700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fc740 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fad2800b790 tx=0x7fad28009220 comp rx=0 tx=0).stop 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 shutdown_connections 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fad240692a0 0x7fad2406b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad380fb560 0x7fad380fc740 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 --2- 172.21.15.73:0/2847917113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad38102fe0 0x7fad380fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 >> 172.21.15.73:0/2847917113 conn(0x7fad38071500 msgr2=0x7fad380fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 shutdown_connections 2024-02-10T06:41:27.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:27.236+0000 7fad40287700 1 -- 172.21.15.73:0/2847917113 wait complete. 2024-02-10T06:41:27.908 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:27 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2847917113' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:41:27.908 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:27.932 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-10T06:41:27.932 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-10T06:41:27.935 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi181.front.sepia.ceph.com 2024-02-10T06:41:27.936 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-10T06:41:27.936 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:27.936 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-10T06:41:27.936 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-10T06:41:27.936 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-10T06:41:27.936 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-10T06:41:27.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:27.936 DEBUG:teuthology.orchestra.run.smithi181:> ip addr 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout: inet 127.0.0.1/8 scope host lo 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 ::1/128 scope host 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:41:27.955 INFO:teuthology.orchestra.run.smithi181.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:d9:39:10 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: altname enp7s0f0 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:d9:39:11 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: altname enp7s0f1 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:88:7b:9c brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:88:7b:9d brd ff:ff:ff:ff:ff:ff 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: inet 172.21.15.181/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft 42293sec preferred_lft 42293sec 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::ec4:7aff:fe88:7b9d/64 scope link 2024-02-10T06:41:27.956 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:41:27.957 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-10T06:41:27.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec bash -c ' 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link add name ceph-brx type bridge 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip addr flush dev ceph-brx 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set ceph-brx up 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-10T06:41:27.957 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-02-10T06:41:28.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:27 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2847917113' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:41:28.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:28.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.137 DEBUG:teuthology.orchestra.run.smithi181:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-10T06:41:28.163 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-02-10T06:41:28.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.165 DEBUG:teuthology.orchestra.run.smithi181:> route 2024-02-10T06:41:28.241 INFO:teuthology.orchestra.run.smithi181.stdout:Kernel IP routing table 2024-02-10T06:41:28.242 INFO:teuthology.orchestra.run.smithi181.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-10T06:41:28.242 INFO:teuthology.orchestra.run.smithi181.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-10T06:41:28.242 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-10T06:41:28.242 INFO:teuthology.orchestra.run.smithi181.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-10T06:41:28.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.242 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec bash -c ' 2024-02-10T06:41:28.242 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-02-10T06:41:28.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-10T06:41:28.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-10T06:41:28.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-10T06:41:28.242 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-02-10T06:41:28.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.524 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-02-10T06:41:28.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.539 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list-id 2024-02-10T06:41:28.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.595 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec bash -c ' 2024-02-10T06:41:28.595 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-02-10T06:41:28.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-10T06:41:28.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-10T06:41:28.595 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-02-10T06:41:28.680 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-10T06:41:28.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.680 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec bash -c ' 2024-02-10T06:41:28.680 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-02-10T06:41:28.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-10T06:41:28.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-10T06:41:28.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-10T06:41:28.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-10T06:41:28.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-10T06:41:28.681 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-02-10T06:41:28.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.784 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec bash -c ' 2024-02-10T06:41:28.784 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-02-10T06:41:28.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set brx.0 up 2024-02-10T06:41:28.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set dev brx.0 master ceph-brx 2024-02-10T06:41:28.784 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-02-10T06:41:28.828 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:41:28.828 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:28.842 INFO:teuthology.orchestra.run.smithi181.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-10T06:41:28.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:41:28.842 DEBUG:teuthology.orchestra.run.smithi181:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:28.897 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-10T06:41:28.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:28.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:41:28.962 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:28 smithi181 ceph-mon[36721]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-10T06:41:28.992 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:41:28.992 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-10T06:41:28.993 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-02-10T06:41:29.109 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:41:29.109 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-10T06:41:29.109 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-02-10T06:41:29.109 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.73:3300,172.21.15.181:3300" 2024-02-10T06:41:29.109 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.73:3300,172.21.15.181:3300" 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.73:6789,172.21.15.181:6789" 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.73:6789,172.21.15.181:6789:/ 2024-02-10T06:41:29.114 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=3c2b87a2-c7de-11ee-95b8-87774f69a715" will pass to kernel 2024-02-10T06:41:29.115 INFO:teuthology.orchestra.run.smithi181.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-10T06:41:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: You just mounted a file system that supports labels which does not 2024-02-10T06:41:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-10T06:41:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: applications will generate AVC messages and not be allowed access to 2024-02-10T06:41:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-10T06:41:29.117 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-10T06:41:29.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:29.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:29.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:28 smithi073 ceph-mon[29063]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-10T06:41:29.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:29.185 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:41:29.240 INFO:teuthology.orchestra.run.smithi181.stdout:client24275 2024-02-10T06:41:29.240 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:41:29.240 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:41:29.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:29.296 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:41:29.352 INFO:teuthology.orchestra.run.smithi181.stdout:client24275 2024-02-10T06:41:29.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:29.352 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:41:29.408 INFO:teuthology.orchestra.run.smithi181.stdout:client24275 2024-02-10T06:41:29.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:41:29.408 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715.client24275/status') 2024-02-10T06:41:29.475 INFO:teuthology.orchestra.run.smithi181.stdout:instance: client.24275 (3)192.168.0.1:0/805294393 2024-02-10T06:41:29.476 INFO:teuthology.orchestra.run.smithi181.stdout:blocklisted: false 2024-02-10T06:41:29.476 INFO:teuthology.orchestra.run.smithi181.stderr:0+1 records in 2024-02-10T06:41:29.476 INFO:teuthology.orchestra.run.smithi181.stderr:0+1 records out 2024-02-10T06:41:29.476 INFO:teuthology.orchestra.run.smithi181.stderr:69 bytes copied, 2.0311e-05 s, 3.4 MB/s 2024-02-10T06:41:29.477 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:41:29.489 INFO:teuthology.task.print:**** done client 2024-02-10T06:41:29.489 INFO:teuthology.run_tasks:Running task parallel... 2024-02-10T06:41:29.529 INFO:teuthology.task.parallel:starting parallel... 2024-02-10T06:41:29.530 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T06:41:29.530 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:41:29.531 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:41:29.531 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-02-10T06:41:29.531 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T06:41:29.531 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-10T06:41:29.534 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:29.535 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-10T06:41:29.535 INFO:tasks.workunit:timeout=3h 2024-02-10T06:41:29.535 INFO:tasks.workunit:cleanup=True 2024-02-10T06:41:29.535 DEBUG:teuthology.orchestra.run.smithi073:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:29.555 INFO:teuthology.orchestra.run.smithi073.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:29.555 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 48h/72d Inode: 1 Links: 2 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-02-10 06:40:55.864259560 +0000 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-02-10 06:41:24.741669854 +0000 2024-02-10T06:41:29.556 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-02-10 06:40:55.864259560 +0000 2024-02-10T06:41:29.556 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-10T06:41:29.556 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-10T06:41:29.629 DEBUG:teuthology.orchestra.run.smithi181:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:29.644 INFO:teuthology.orchestra.run.smithi181.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:29.644 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-10 06:41:29.624551498 +0000 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-10 06:41:29.624551498 +0000 2024-02-10T06:41:29.645 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-02-10 06:40:55.864259560 +0000 2024-02-10T06:41:29.645 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-10T06:41:29.645 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-10T06:41:29.715 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:29.716 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:29.732 INFO:tasks.workunit.client.0.smithi073.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-10T06:41:29.771 INFO:tasks.workunit.client.1.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-10T06:41:29.815 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:31.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:30 smithi181 ceph-mon[36721]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-10T06:41:31.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:30 smithi073 ceph-mon[29063]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-10T06:41:31.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 -- 172.21.15.73:0/4237941983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fcef0 msgr2=0x7f46d00fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:31.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 --2- 172.21.15.73:0/4237941983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fcef0 0x7f46d00fd320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f46cc004940 tx=0x7f46cc009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:31.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 -- 172.21.15.73:0/4237941983 shutdown_connections 2024-02-10T06:41:31.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 --2- 172.21.15.73:0/4237941983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d00fe0e0 0x7f46d00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 --2- 172.21.15.73:0/4237941983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fcef0 0x7f46d00fd320 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 -- 172.21.15.73:0/4237941983 >> 172.21.15.73:0/4237941983 conn(0x7f46d00f8620 msgr2=0x7f46d00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 -- 172.21.15.73:0/4237941983 shutdown_connections 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.581+0000 7f46d8387700 1 -- 172.21.15.73:0/4237941983 wait complete. 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d8387700 1 Processor -- start 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d8387700 1 -- start start 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d8387700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d00fcef0 0x7f46d0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d8387700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fe0e0 0x7f46d00702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d8387700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00708e0 con 0x7f46d00fe0e0 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d8387700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0070a50 con 0x7f46d00fcef0 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d5922700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fe0e0 0x7f46d00702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d5922700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fe0e0 0x7f46d00702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58196/0 (socket says 172.21.15.73:58196) 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d5922700 1 -- 172.21.15.73:0/374721284 learned_addr learned my addr 172.21.15.73:0/374721284 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d6123700 1 --2- 172.21.15.73:0/374721284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d00fcef0 0x7f46d0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:31.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d5922700 1 -- 172.21.15.73:0/374721284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d00fcef0 msgr2=0x7f46d0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d5922700 1 --2- 172.21.15.73:0/374721284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d00fcef0 0x7f46d0071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.582+0000 7f46d5922700 1 -- 172.21.15.73:0/374721284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc005040 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.583+0000 7f46d5922700 1 --2- 172.21.15.73:0/374721284 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f46d00fe0e0 0x7f46d00702e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f46c0002a50 tx=0x7f46c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.583+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c0007cc0 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.583+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46c000f040 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.583+0000 7f46d8387700 1 -- 172.21.15.73:0/374721284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d0070d30 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.583+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c000da30 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.583+0000 7f46d8387700 1 -- 172.21.15.73:0/374721284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d01159a0 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.584+0000 7f46d8387700 1 -- 172.21.15.73:0/374721284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d004bed0 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.584+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f46c000db90 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.585+0000 7f46c6ffd700 1 --2- 172.21.15.73:0/374721284 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f46bc0692a0 0x7f46bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.585+0000 7f46d6123700 1 --2- 172.21.15.73:0/374721284 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f46bc0692a0 0x7f46bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.585+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f46c0086f90 con 0x7f46d00fe0e0 2024-02-10T06:41:31.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.585+0000 7f46d6123700 1 --2- 172.21.15.73:0/374721284 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f46bc0692a0 0x7f46bc06b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00b840 tx=0x7f46cc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:31.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.589+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f46c0056c80 con 0x7f46d00fe0e0 2024-02-10T06:41:31.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.773+0000 7f46d8387700 1 -- 172.21.15.73:0/374721284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f46d0105a30 con 0x7f46d00fe0e0 2024-02-10T06:41:31.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:31.773+0000 7f46c6ffd700 1 -- 172.21.15.73:0/374721284 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7f46c0056810 con 0x7f46d00fe0e0 2024-02-10T06:41:31.776 INFO:teuthology.orchestra.run.smithi073.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-02-10T06:41:32.572 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:41:32.573 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:41:32.573 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T06:41:32.868 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:33.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:32 smithi181 ceph-mon[36721]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:41:33.099 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:32 smithi073 ceph-mon[29063]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:41:34.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2537376160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 msgr2=0x7f0b340fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:34.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2537376160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b340fd420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0b24004940 tx=0x7f0b24009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:34.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2537376160 shutdown_connections 2024-02-10T06:41:34.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2537376160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b340fd420 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:34.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2537376160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b340fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2537376160 >> 172.21.15.73:0/2537376160 conn(0x7f0b340f7520 msgr2=0x7f0b340f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2537376160 shutdown_connections 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.611+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2537376160 wait complete. 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b3bafd700 1 Processor -- start 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b3bafd700 1 -- start start 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b3bafd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b341105c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b3bafd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b34112b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b3bafd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34066cf0 con 0x7f0b340fcfb0 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b3bafd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34066e30 con 0x7f0b340fbe40 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39098700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b34112b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39899700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b341105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39899700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b341105c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.73:36392/0 (socket says 172.21.15.73:36392) 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39098700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b34112b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58218/0 (socket says 172.21.15.73:58218) 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39899700 1 -- 172.21.15.73:0/2996997121 learned_addr learned my addr 172.21.15.73:0/2996997121 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:34.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39899700 1 -- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 msgr2=0x7f0b34112b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:34.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39899700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b34112b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:34.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.612+0000 7f0b39899700 1 -- 172.21.15.73:0/2996997121 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24005040 con 0x7f0b340fbe40 2024-02-10T06:41:34.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.613+0000 7f0b39899700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b341105c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000eb40 tx=0x7f0b3000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.613+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b30007cc0 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.613+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b3000f040 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.613+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b3000ca10 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.613+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b34113110 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.613+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b34113600 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.614+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3404bed0 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.614+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0b3000cc80 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.615+0000 7f0b2a7fc700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0b200692a0 0x7f0b2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.615+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0b30086e70 con 0x7f0b340fbe40 2024-02-10T06:41:34.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.615+0000 7f0b39098700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0b200692a0 0x7f0b2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:34.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.615+0000 7f0b39098700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0b200692a0 0x7f0b2006b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009b30 tx=0x7f0b24009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:34.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.619+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0b3004dbd0 con 0x7f0b340fbe40 2024-02-10T06:41:34.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.807+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f0b34104930 con 0x7f0b340fbe40 2024-02-10T06:41:34.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.812+0000 7f0b2a7fc700 1 -- 172.21.15.73:0/2996997121 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f0b300530e0 con 0x7f0b340fbe40 2024-02-10T06:41:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0b200692a0 msgr2=0x7f0b2006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0b200692a0 0x7f0b2006b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009b30 tx=0x7f0b24009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 msgr2=0x7f0b341105c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b341105c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000eb40 tx=0x7f0b3000b040 comp rx=0 tx=0).stop 2024-02-10T06:41:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 shutdown_connections 2024-02-10T06:41:34.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0b200692a0 0x7f0b2006b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:34.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b340fbe40 0x7f0b341105c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:34.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 --2- 172.21.15.73:0/2996997121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b340fcfb0 0x7f0b34112b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:34.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 >> 172.21.15.73:0/2996997121 conn(0x7f0b340f7520 msgr2=0x7f0b34100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:34.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 shutdown_connections 2024-02-10T06:41:34.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:34.814+0000 7f0b3bafd700 1 -- 172.21.15.73:0/2996997121 wait complete. 2024-02-10T06:41:34.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:34 smithi073 ceph-mon[29063]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:41:35.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:34 smithi181 ceph-mon[36721]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:41:35.670 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T06:41:35.944 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:37.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:36 smithi181 ceph-mon[36721]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:41:37.120 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:36 smithi073 ceph-mon[29063]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 -- 172.21.15.73:0/664174441 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fcfb0 msgr2=0x7fe32c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 --2- 172.21.15.73:0/664174441 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fcfb0 0x7fe32c0fd420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe31c004940 tx=0x7fe31c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 -- 172.21.15.73:0/664174441 shutdown_connections 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 --2- 172.21.15.73:0/664174441 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fcfb0 0x7fe32c0fd420 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 --2- 172.21.15.73:0/664174441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe32c0fbe40 0x7fe32c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 -- 172.21.15.73:0/664174441 >> 172.21.15.73:0/664174441 conn(0x7fe32c0f74e0 msgr2=0x7fe32c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 -- 172.21.15.73:0/664174441 shutdown_connections 2024-02-10T06:41:37.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.806+0000 7fe333ffe700 1 -- 172.21.15.73:0/664174441 wait complete. 2024-02-10T06:41:37.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe333ffe700 1 Processor -- start 2024-02-10T06:41:37.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe333ffe700 1 -- start start 2024-02-10T06:41:37.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe333ffe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 0x7fe32c10c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:37.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe333ffe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe32c0fcfb0 0x7fe32c10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:37.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe333ffe700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c10d420 con 0x7fe32c0fbe40 2024-02-10T06:41:37.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe333ffe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c10dc60 con 0x7fe32c0fcfb0 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331d9a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 0x7fe32c10c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331d9a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 0x7fe32c10c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58232/0 (socket says 172.21.15.73:58232) 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331d9a700 1 -- 172.21.15.73:0/2472679795 learned_addr learned my addr 172.21.15.73:0/2472679795 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331599700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe32c0fcfb0 0x7fe32c10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331d9a700 1 -- 172.21.15.73:0/2472679795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe32c0fcfb0 msgr2=0x7fe32c10cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331d9a700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe32c0fcfb0 0x7fe32c10cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.807+0000 7fe331d9a700 1 -- 172.21.15.73:0/2472679795 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c005040 con 0x7fe32c0fbe40 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.808+0000 7fe331d9a700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 0x7fe32c10c850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe32800b790 tx=0x7fe328009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.808+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe32800f6f0 con 0x7fe32c0fbe40 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.808+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c10df40 con 0x7fe32c0fbe40 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.808+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe32800fd20 con 0x7fe32c0fbe40 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.808+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3280103a0 con 0x7fe32c0fbe40 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.808+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c10e480 con 0x7fe32c0fbe40 2024-02-10T06:41:37.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.809+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c04bed0 con 0x7fe32c0fbe40 2024-02-10T06:41:37.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.809+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe328011400 con 0x7fe32c0fbe40 2024-02-10T06:41:37.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.810+0000 7fe3227fc700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe3180692a0 0x7fe31806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:37.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.810+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe328086ee0 con 0x7fe32c0fbe40 2024-02-10T06:41:37.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.810+0000 7fe331599700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe3180692a0 0x7fe31806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:37.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.810+0000 7fe331599700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe3180692a0 0x7fe31806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00b3b0 tx=0x7fe31c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:37.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:37.818+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe328056cf0 con 0x7fe32c0fbe40 2024-02-10T06:41:38.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.028+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fe32c1049b0 con 0x7fe32c0fbe40 2024-02-10T06:41:38.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.028+0000 7fe3227fc700 1 -- 172.21.15.73:0/2472679795 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fe328056880 con 0x7fe32c0fbe40 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe3180692a0 msgr2=0x7fe31806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe3180692a0 0x7fe31806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00b3b0 tx=0x7fe31c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 msgr2=0x7fe32c10c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 0x7fe32c10c850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe32800b790 tx=0x7fe328009250 comp rx=0 tx=0).stop 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 shutdown_connections 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fe3180692a0 0x7fe31806b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:38.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe32c0fbe40 0x7fe32c10c850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:38.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 --2- 172.21.15.73:0/2472679795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe32c0fcfb0 0x7fe32c10cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:38.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 >> 172.21.15.73:0/2472679795 conn(0x7fe32c0f74e0 msgr2=0x7fe32c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:38.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.030+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 shutdown_connections 2024-02-10T06:41:38.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:38.031+0000 7fe333ffe700 1 -- 172.21.15.73:0/2472679795 wait complete. 2024-02-10T06:41:38.923 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T06:41:39.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:38 smithi181 ceph-mon[36721]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-10T06:41:39.120 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:38 smithi073 ceph-mon[29063]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-10T06:41:39.192 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:41.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 -- 172.21.15.73:0/73391148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 msgr2=0x7fcb14102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 --2- 172.21.15.73:0/73391148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb14102530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcb08004940 tx=0x7fcb08009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 -- 172.21.15.73:0/73391148 shutdown_connections 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 --2- 172.21.15.73:0/73391148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb14102530 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 --2- 172.21.15.73:0/73391148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb14066c90 0x7fcb140ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 -- 172.21.15.73:0/73391148 >> 172.21.15.73:0/73391148 conn(0x7fcb140f7510 msgr2=0x7fcb140f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.022+0000 7fcb199db700 1 -- 172.21.15.73:0/73391148 shutdown_connections 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 -- 172.21.15.73:0/73391148 wait complete. 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 Processor -- start 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 -- start start 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb14066c90 0x7fcb1410d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb1410de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1410e410 con 0x7fcb141000b0 2024-02-10T06:41:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.023+0000 7fcb199db700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1410ecb0 con 0x7fcb14066c90 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb1410de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb1410de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46752/0 (socket says 172.21.15.73:46752) 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb12ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb14066c90 0x7fcb1410d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 -- 172.21.15.73:0/2738466966 learned_addr learned my addr 172.21.15.73:0/2738466966 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 -- 172.21.15.73:0/2738466966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb14066c90 msgr2=0x7fcb1410d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb14066c90 0x7fcb1410d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 -- 172.21.15.73:0/2738466966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08005040 con 0x7fcb141000b0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb127fc700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb1410de10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcb080093b0 tx=0x7fcb08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0800e040 con 0x7fcb141000b0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1410ef30 con 0x7fcb141000b0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb0800bcc0 con 0x7fcb141000b0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0801c630 con 0x7fcb141000b0 2024-02-10T06:41:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.024+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1410f470 con 0x7fcb141000b0 2024-02-10T06:41:41.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.026+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb14107a60 con 0x7fcb141000b0 2024-02-10T06:41:41.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.026+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcb08013980 con 0x7fcb141000b0 2024-02-10T06:41:41.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.026+0000 7fcafb7fe700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcafc0692a0 0x7fcafc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:41.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.027+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcb08088600 con 0x7fcb141000b0 2024-02-10T06:41:41.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.027+0000 7fcb12ffd700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcafc0692a0 0x7fcafc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:41.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.027+0000 7fcb12ffd700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcafc0692a0 0x7fcafc06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009fd0 tx=0x7fcb0400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.031+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcb080582d0 con 0x7fcb141000b0 2024-02-10T06:41:41.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:40 smithi181 ceph-mon[36721]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:41:41.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:40 smithi073 ceph-mon[29063]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:41:41.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.246+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fcb14105a20 con 0x7fcb141000b0 2024-02-10T06:41:41.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.246+0000 7fcafb7fe700 1 -- 172.21.15.73:0/2738466966 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fcb08057e60 con 0x7fcb141000b0 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcafc0692a0 msgr2=0x7fcafc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcafc0692a0 0x7fcafc06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009fd0 tx=0x7fcb0400b040 comp rx=0 tx=0).stop 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 msgr2=0x7fcb1410de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb1410de10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcb080093b0 tx=0x7fcb08009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 shutdown_connections 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fcafc0692a0 0x7fcafc06b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb14066c90 0x7fcb1410d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 --2- 172.21.15.73:0/2738466966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb141000b0 0x7fcb1410de10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 >> 172.21.15.73:0/2738466966 conn(0x7fcb140f7510 msgr2=0x7fcb140f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 shutdown_connections 2024-02-10T06:41:41.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:41.249+0000 7fcb199db700 1 -- 172.21.15.73:0/2738466966 wait complete. 2024-02-10T06:41:42.113 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T06:41:42.379 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:43.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:42 smithi181 ceph-mon[36721]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:41:43.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:42 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:43.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:42 smithi073 ceph-mon[29063]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:41:43.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:42 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/2722719470 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 msgr2=0x7f1d540fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/2722719470 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d540fd420 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1d48004940 tx=0x7f1d48009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/2722719470 shutdown_connections 2024-02-10T06:41:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/2722719470 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d540fd420 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/2722719470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d540fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/2722719470 >> 172.21.15.73:0/2722719470 conn(0x7f1d540f7520 msgr2=0x7f1d540f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/2722719470 shutdown_connections 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/2722719470 wait complete. 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 Processor -- start 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.269+0000 7f1d5a9b5700 1 -- start start 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d5a9b5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d5410c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d5a9b5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d5410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d5a9b5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5410d2e0 con 0x7f1d540fcfb0 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d5a9b5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5410db80 con 0x7f1d540fbe40 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d537fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d5410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d5410c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d5410c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.73:40912/0 (socket says 172.21.15.73:40912) 2024-02-10T06:41:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d537fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d5410cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46770/0 (socket says 172.21.15.73:46770) 2024-02-10T06:41:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 -- 172.21.15.73:0/1182951882 learned_addr learned my addr 172.21.15.73:0/1182951882 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 -- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 msgr2=0x7f1d5410cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d5410cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 -- 172.21.15.73:0/1182951882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d48005040 con 0x7f1d540fbe40 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d53fff700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d5410c7a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400eb40 tx=0x7f1d4400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.270+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d44007cc0 con 0x7f1d540fbe40 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.271+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5410de00 con 0x7f1d540fbe40 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.271+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d4400f040 con 0x7f1d540fbe40 2024-02-10T06:41:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.271+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d4400ca10 con 0x7f1d540fbe40 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.271+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5410e340 con 0x7f1d540fbe40 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.272+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5404bed0 con 0x7f1d540fbe40 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.272+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1d4400cc80 con 0x7f1d540fbe40 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.272+0000 7f1d50ff9700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1d3c06b4b0 0x7f1d3c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.273+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1d44086dd0 con 0x7f1d540fbe40 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.273+0000 7f1d537fe700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1d3c06b4b0 0x7f1d3c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:44.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.273+0000 7f1d537fe700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1d3c06b4b0 0x7f1d3c06d960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009b30 tx=0x7f1d48009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:44.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.277+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1d4404dbd0 con 0x7f1d540fbe40 2024-02-10T06:41:44.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.505+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d5410e760 con 0x7f1d3c06b4b0 2024-02-10T06:41:44.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.510+0000 7f1d50ff9700 1 -- 172.21.15.73:0/1182951882 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f1d5410e760 con 0x7f1d3c06b4b0 2024-02-10T06:41:44.511 INFO:teuthology.orchestra.run.smithi073.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:44.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1d3c06b4b0 msgr2=0x7f1d3c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:44.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1d3c06b4b0 0x7f1d3c06d960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009b30 tx=0x7f1d48009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:44.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 msgr2=0x7f1d5410c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:44.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d5410c7a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400eb40 tx=0x7f1d4400b040 comp rx=0 tx=0).stop 2024-02-10T06:41:44.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 shutdown_connections 2024-02-10T06:41:44.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f1d3c06b4b0 0x7f1d3c06d960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:44.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d540fbe40 0x7f1d5410c7a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:44.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 --2- 172.21.15.73:0/1182951882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1d540fcfb0 0x7f1d5410cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:44.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 >> 172.21.15.73:0/1182951882 conn(0x7f1d540f7520 msgr2=0x7f1d54100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:44.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 shutdown_connections 2024-02-10T06:41:44.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:44.512+0000 7f1d5a9b5700 1 -- 172.21.15.73:0/1182951882 wait complete. 2024-02-10T06:41:45.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:44 smithi181 ceph-mon[36721]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-10T06:41:45.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:44 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:45.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:44 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:45.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:44 smithi073 ceph-mon[29063]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-10T06:41:45.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:44 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:45.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:44 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:45.364 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:41:45.364 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:41:45.364 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-10T06:41:45.657 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:41:46.043 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:45 smithi073 ceph-mon[29063]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:46.043 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:45 smithi073 ceph-mon[29063]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:46.044 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:45 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:46.044 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:45 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:46.044 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:45 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:46.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:45 smithi181 ceph-mon[36721]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:46.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:45 smithi181 ceph-mon[36721]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:46.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:45 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:46.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:45 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:46.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:45 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:41:47.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:46 smithi181 ceph-mon[36721]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-10T06:41:47.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:46 smithi181 ceph-mon[36721]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:46 smithi073 ceph-mon[29063]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-10T06:41:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:46 smithi073 ceph-mon[29063]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:41:47.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 -- 172.21.15.73:0/4263508248 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f82100fb300 msgr2=0x7f82100fb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:47.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 --2- 172.21.15.73:0/4263508248 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f82100fb300 0x7f82100fb710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8200004940 tx=0x7f8200009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:47.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 -- 172.21.15.73:0/4263508248 shutdown_connections 2024-02-10T06:41:47.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 --2- 172.21.15.73:0/4263508248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82100fbce0 0x7f8210102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:47.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 --2- 172.21.15.73:0/4263508248 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f82100fb300 0x7f82100fb710 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:47.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 -- 172.21.15.73:0/4263508248 >> 172.21.15.73:0/4263508248 conn(0x7f82100f6ca0 msgr2=0x7f82100f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:47.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.647+0000 7f8216129700 1 -- 172.21.15.73:0/4263508248 shutdown_connections 2024-02-10T06:41:47.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.648+0000 7f8216129700 1 -- 172.21.15.73:0/4263508248 wait complete. 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.648+0000 7f8216129700 1 Processor -- start 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f8216129700 1 -- start start 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f8216129700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82100fb300 0x7f821010d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f8216129700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 0x7f821010eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f8216129700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821010e1a0 con 0x7f821010dce0 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f820effd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 0x7f821010eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f820effd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 0x7f821010eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46788/0 (socket says 172.21.15.73:46788) 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f820effd700 1 -- 172.21.15.73:0/2313144395 learned_addr learned my addr 172.21.15.73:0/2313144395 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:47.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.649+0000 7f8216129700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821010e2e0 con 0x7f82100fb300 2024-02-10T06:41:47.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f820f7fe700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82100fb300 0x7f821010d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:47.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f820effd700 1 -- 172.21.15.73:0/2313144395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82100fb300 msgr2=0x7f821010d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:47.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f820effd700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82100fb300 0x7f821010d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:47.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f820effd700 1 -- 172.21.15.73:0/2313144395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200005040 con 0x7f821010dce0 2024-02-10T06:41:47.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f820effd700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 0x7f821010eb80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f820400eb40 tx=0x7f820400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:47.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8204007cc0 con 0x7f821010dce0 2024-02-10T06:41:47.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.650+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821010f180 con 0x7f821010dce0 2024-02-10T06:41:47.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.651+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821010f690 con 0x7f821010dce0 2024-02-10T06:41:47.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.652+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f820400f040 con 0x7f821010dce0 2024-02-10T06:41:47.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.652+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f820400c9d0 con 0x7f821010dce0 2024-02-10T06:41:47.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.653+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8210107ab0 con 0x7f821010dce0 2024-02-10T06:41:47.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.654+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f820400cb30 con 0x7f821010dce0 2024-02-10T06:41:47.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.654+0000 7f81f7fff700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f81f80692b0 0x7f81f806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:47.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.655+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8204086be0 con 0x7f821010dce0 2024-02-10T06:41:47.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.655+0000 7f820f7fe700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f81f80692b0 0x7f81f806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:47.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.656+0000 7f820f7fe700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f81f80692b0 0x7f81f806b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8200009fd0 tx=0x7f8200009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:47.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.663+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f820404dbd0 con 0x7f821010dce0 2024-02-10T06:41:47.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.955+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8210105c20 con 0x7f81f80692b0 2024-02-10T06:41:47.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.956+0000 7f81f7fff700 1 -- 172.21.15.73:0/2313144395 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f8200007de0 con 0x7f81f80692b0 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f81f80692b0 msgr2=0x7f81f806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f81f80692b0 0x7f81f806b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8200009fd0 tx=0x7f8200009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 msgr2=0x7f821010eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 0x7f821010eb80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f820400eb40 tx=0x7f820400b040 comp rx=0 tx=0).stop 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 shutdown_connections 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f81f80692b0 0x7f81f806b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82100fb300 0x7f821010d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 --2- 172.21.15.73:0/2313144395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f821010dce0 0x7f821010eb80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.959+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 >> 172.21.15.73:0/2313144395 conn(0x7f82100f6ca0 msgr2=0x7f8210100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.960+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 shutdown_connections 2024-02-10T06:41:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:47.960+0000 7f8216129700 1 -- 172.21.15.73:0/2313144395 wait complete. 2024-02-10T06:41:47.973 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-02-10T06:41:48.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 -- 172.21.15.73:0/3550384071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900a4220 msgr2=0x7f01900ab380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:48.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 --2- 172.21.15.73:0/3550384071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900a4220 0x7f01900ab380 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0188004940 tx=0x7f0188009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:48.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 -- 172.21.15.73:0/3550384071 shutdown_connections 2024-02-10T06:41:48.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 --2- 172.21.15.73:0/3550384071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900a4220 0x7f01900ab380 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0188004940 tx=0x7f0188009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:48.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 --2- 172.21.15.73:0/3550384071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 -- 172.21.15.73:0/3550384071 >> 172.21.15.73:0/3550384071 conn(0x7f019009f090 msgr2=0x7f01900a14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 -- 172.21.15.73:0/3550384071 shutdown_connections 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.119+0000 7f019fcc4700 1 -- 172.21.15.73:0/3550384071 wait complete. 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019fcc4700 1 Processor -- start 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019fcc4700 1 -- start start 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019fcc4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019fcc4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900b48d0 0x7f01900b9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:48.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019fcc4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01900b4db0 con 0x7f01900b48d0 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019fcc4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01900b4f20 con 0x7f01900a3840 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019da60700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019da60700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.73:40940/0 (socket says 172.21.15.73:40940) 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019da60700 1 -- 172.21.15.73:0/704407943 learned_addr learned my addr 172.21.15.73:0/704407943 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.120+0000 7f019d25f700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900b48d0 0x7f01900b9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.121+0000 7f019da60700 1 -- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900b48d0 msgr2=0x7f01900b9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.121+0000 7f019da60700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900b48d0 0x7f01900b9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.121+0000 7f019da60700 1 -- 172.21.15.73:0/704407943 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188005040 con 0x7f01900a3840 2024-02-10T06:41:48.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.122+0000 7f019da60700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900b4390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f019400eb40 tx=0x7f019400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:48.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.123+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0194007cc0 con 0x7f01900a3840 2024-02-10T06:41:48.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.123+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01900b9ed0 con 0x7f01900a3840 2024-02-10T06:41:48.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.123+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01900ba490 con 0x7f01900a3840 2024-02-10T06:41:48.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.123+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f019400f040 con 0x7f01900a3840 2024-02-10T06:41:48.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.123+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019400ca90 con 0x7f01900a3840 2024-02-10T06:41:48.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.124+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c0052e0 con 0x7f01900a3840 2024-02-10T06:41:48.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.126+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f019400cbf0 con 0x7f01900a3840 2024-02-10T06:41:48.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.126+0000 7f018e7fc700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0184069490 0x7f018406b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:48.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.127+0000 7f019d25f700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0184069490 0x7f018406b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:48.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.127+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0194086e10 con 0x7f01900a3840 2024-02-10T06:41:48.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.127+0000 7f019d25f700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0184069490 0x7f018406b940 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f018800b0d0 tx=0x7f0188009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:48.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.134+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f01940530d0 con 0x7f01900a3840 2024-02-10T06:41:48.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.468+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f017c000bf0 con 0x7f0184069490 2024-02-10T06:41:48.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.469+0000 7f018e7fc700 1 -- 172.21.15.73:0/704407943 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f017c000bf0 con 0x7f0184069490 2024-02-10T06:41:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.472+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0184069490 msgr2=0x7f018406b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.472+0000 7f019fcc4700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0184069490 0x7f018406b940 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f018800b0d0 tx=0x7f0188009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.472+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 msgr2=0x7f01900b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.472+0000 7f019fcc4700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900b4390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f019400eb40 tx=0x7f019400b040 comp rx=0 tx=0).stop 2024-02-10T06:41:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 shutdown_connections 2024-02-10T06:41:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f0184069490 0x7f018406b940 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01900a3840 0x7f01900b4390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 --2- 172.21.15.73:0/704407943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01900b48d0 0x7f01900b9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 >> 172.21.15.73:0/704407943 conn(0x7f019009f090 msgr2=0x7f01900a79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 shutdown_connections 2024-02-10T06:41:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.473+0000 7f019fcc4700 1 -- 172.21.15.73:0/704407943 wait complete. 2024-02-10T06:41:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.631+0000 7f762cd5f700 1 -- 172.21.15.73:0/3833546737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 msgr2=0x7f76280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.631+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3833546737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f76280fd420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f761c004880 tx=0x7f761c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.631+0000 7f762cd5f700 1 -- 172.21.15.73:0/3833546737 shutdown_connections 2024-02-10T06:41:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.631+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3833546737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f76280fd420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.631+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3833546737 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f76280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.631+0000 7f762cd5f700 1 -- 172.21.15.73:0/3833546737 >> 172.21.15.73:0/3833546737 conn(0x7f76280f7520 msgr2=0x7f76280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.632+0000 7f762cd5f700 1 -- 172.21.15.73:0/3833546737 shutdown_connections 2024-02-10T06:41:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.632+0000 7f762cd5f700 1 -- 172.21.15.73:0/3833546737 wait complete. 2024-02-10T06:41:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.633+0000 7f762cd5f700 1 Processor -- start 2024-02-10T06:41:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.633+0000 7f762cd5f700 1 -- start start 2024-02-10T06:41:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762cd5f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f762810b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762cd5f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f762810bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f762810b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f762810b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51900/0 (socket says 172.21.15.73:51900) 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762cd5f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762810c280 con 0x7f76280fbe40 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762cd5f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762810cb20 con 0x7f76280fcfb0 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762659c700 1 -- 172.21.15.73:0/3971540513 learned_addr learned my addr 172.21.15.73:0/3971540513 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f7625d9b700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f762810bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762659c700 1 -- 172.21.15.73:0/3971540513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 msgr2=0x7f762810bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762659c700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f762810bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f762659c700 1 -- 172.21.15.73:0/3971540513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c005040 con 0x7f76280fbe40 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.634+0000 7f7625d9b700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f762810bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.635+0000 7f762659c700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f762810b740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76180041d0 tx=0x7f7618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.635+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7618010e80 con 0x7f76280fbe40 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.635+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76180090d0 con 0x7f76280fbe40 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.635+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762810cda0 con 0x7f76280fbe40 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.635+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762810d220 con 0x7f76280fbe40 2024-02-10T06:41:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.636+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7618012700 con 0x7f76280fbe40 2024-02-10T06:41:48.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.637+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762804bed0 con 0x7f76280fbe40 2024-02-10T06:41:48.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.637+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7618018030 con 0x7f76280fbe40 2024-02-10T06:41:48.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.638+0000 7f7616ffd700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7610069440 0x7f761006b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:48.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.638+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7618016070 con 0x7f76280fbe40 2024-02-10T06:41:48.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.638+0000 7f7625d9b700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7610069440 0x7f761006b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:48.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.639+0000 7f7625d9b700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7610069440 0x7f761006b8f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f761c009fd0 tx=0x7f761c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:48.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.643+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7618059930 con 0x7f76280fbe40 2024-02-10T06:41:48.838 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:48 smithi073 ceph-mon[29063]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-10T06:41:48.838 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:48 smithi073 ceph-mon[29063]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:48.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.981+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7628101990 con 0x7f7610069440 2024-02-10T06:41:49.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:48.999+0000 7f7616ffd700 1 -- 172.21.15.73:0/3971540513 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f7628101990 con 0x7f7610069440 2024-02-10T06:41:49.000 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:41:49.000 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (5m) 28s ago 6m 26.7M - 0.25.0 c8568f914cd2 21a01485a256 2024-02-10T06:41:49.000 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (6m) 28s ago 6m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 26dabce1f0be 2024-02-10T06:41:49.000 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi181 smithi181 running (5m) 34s ago 5m 6656k - 17.2.7-505-g72ce7c1d a2a0bd126c96 0ed4bbd7eaf3 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (4m) 28s ago 6m 84.3M - 9.4.7 954c08fa6188 9edb0a6c1f79 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi073.nuvejb smithi073 running (54s) 28s ago 53s 10.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9e00530efbe3 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi073.ykxzap smithi073 running (47s) 28s ago 47s 13.6M - 17.2.7-505-g72ce7c1d a2a0bd126c96 95e7cbe8df86 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi181.cuebex smithi181 running (52s) 34s ago 51s 16.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 e3a8d959df73 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi181.otunfp smithi181 running (44s) 34s ago 44s 13.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9a844ba6a8fb 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.vjivfn smithi073 *:9283 running (7m) 28s ago 7m 481M - 17.2.7-505-g72ce7c1d a2a0bd126c96 df67d3a17ded 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi181.lmidhc smithi181 *:8443,9283 running (5m) 34s ago 5m 433M - 17.2.7-505-g72ce7c1d a2a0bd126c96 0ede8e2f6c22 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (7m) 28s ago 7m 48.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 9743a7ae51ad 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi181 smithi181 running (5m) 34s ago 5m 35.4M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 335b19e9983e 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (6m) 28s ago 6m 16.2M - 1.5.0 0da6a335fe13 f5df22ebae55 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi181 smithi181 *:9100 running (5m) 34s ago 5m 16.3M - 1.5.0 0da6a335fe13 8b2c66b967b3 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (4m) 28s ago 4m 46.8M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 9218e1a18fa5 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (4m) 28s ago 4m 48.8M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 cffadaeafb44 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (3m) 28s ago 3m 46.7M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 c93fd57a90db 2024-02-10T06:41:49.001 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi181 running (3m) 34s ago 3m 44.0M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 ebadff8f8025 2024-02-10T06:41:49.002 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi181 running (2m) 34s ago 2m 44.5M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 56929cb111f3 2024-02-10T06:41:49.002 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi181 running (2m) 34s ago 2m 41.4M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 85003a026df4 2024-02-10T06:41:49.002 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (4m) 28s ago 5m 52.9M - 2.43.0 a07b618ecd1d 2e631f28238b 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7610069440 msgr2=0x7f761006b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7610069440 0x7f761006b8f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f761c009fd0 tx=0x7f761c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 msgr2=0x7f762810b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f762810b740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76180041d0 tx=0x7f7618009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 shutdown_connections 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f7610069440 0x7f761006b8f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76280fbe40 0x7f762810b740 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 --2- 172.21.15.73:0/3971540513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76280fcfb0 0x7f762810bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 >> 172.21.15.73:0/3971540513 conn(0x7f76280f7520 msgr2=0x7f7628100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 shutdown_connections 2024-02-10T06:41:49.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.002+0000 7f762cd5f700 1 -- 172.21.15.73:0/3971540513 wait complete. 2024-02-10T06:41:49.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:48 smithi181 ceph-mon[36721]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-10T06:41:49.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:48 smithi181 ceph-mon[36721]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 -- 172.21.15.73:0/3774324820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 msgr2=0x7f861c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 --2- 172.21.15.73:0/3774324820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c0fd420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f860c004940 tx=0x7f860c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 -- 172.21.15.73:0/3774324820 shutdown_connections 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 --2- 172.21.15.73:0/3774324820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c0fd420 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 --2- 172.21.15.73:0/3774324820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 0x7f861c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 -- 172.21.15.73:0/3774324820 >> 172.21.15.73:0/3774324820 conn(0x7f861c0f7500 msgr2=0x7f861c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 -- 172.21.15.73:0/3774324820 shutdown_connections 2024-02-10T06:41:49.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.147+0000 7f8624734700 1 -- 172.21.15.73:0/3774324820 wait complete. 2024-02-10T06:41:49.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.148+0000 7f8624734700 1 Processor -- start 2024-02-10T06:41:49.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.148+0000 7f8624734700 1 -- start start 2024-02-10T06:41:49.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.148+0000 7f8624734700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 0x7f861c10d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8624734700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c10ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8624734700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c10e3f0 con 0x7f861c0fcfb0 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8621ccf700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c10ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8621ccf700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c10ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51914/0 (socket says 172.21.15.73:51914) 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f86224d0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 0x7f861c10d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8621ccf700 1 -- 172.21.15.73:0/3425506224 learned_addr learned my addr 172.21.15.73:0/3425506224 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:49.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c10ec90 con 0x7f861c0fbe40 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8621ccf700 1 -- 172.21.15.73:0/3425506224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 msgr2=0x7f861c10d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8621ccf700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 0x7f861c10d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f8621ccf700 1 -- 172.21.15.73:0/3425506224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c005040 con 0x7f861c0fcfb0 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.149+0000 7f86224d0700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 0x7f861c10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.150+0000 7f8621ccf700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c10ddf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f860c000c00 tx=0x7f860c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.150+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f860c021070 con 0x7f861c0fcfb0 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.150+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f860c01c400 con 0x7f861c0fcfb0 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.150+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f860c00e070 con 0x7f861c0fcfb0 2024-02-10T06:41:49.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.150+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c10ef10 con 0x7f861c0fcfb0 2024-02-10T06:41:49.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.151+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c10f3f0 con 0x7f861c0fcfb0 2024-02-10T06:41:49.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.153+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f860c01ca50 con 0x7f861c0fcfb0 2024-02-10T06:41:49.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.153+0000 7f8612ffd700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f8608069370 0x7f860806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:49.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.153+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f860c088380 con 0x7f861c0fcfb0 2024-02-10T06:41:49.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.154+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c04bed0 con 0x7f861c0fcfb0 2024-02-10T06:41:49.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.154+0000 7f86224d0700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f8608069370 0x7f860806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:49.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.154+0000 7f86224d0700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f8608069370 0x7f860806b820 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8618009b80 tx=0x7f861800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:49.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.163+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f860c0580d0 con 0x7f861c0fcfb0 2024-02-10T06:41:49.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.529+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f861c105a40 con 0x7f861c0fcfb0 2024-02-10T06:41:49.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.529+0000 7f8612ffd700 1 -- 172.21.15.73:0/3425506224 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f860c057c60 con 0x7f861c0fcfb0 2024-02-10T06:41:49.530 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-02-10T06:41:49.530 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-02-10T06:41:49.530 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:41:49.530 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:41:49.530 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 6 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 4 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 14 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-02-10T06:41:49.531 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.533+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f8608069370 msgr2=0x7f860806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.533+0000 7f8624734700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f8608069370 0x7f860806b820 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8618009b80 tx=0x7f861800b040 comp rx=0 tx=0).stop 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 msgr2=0x7f861c10ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c10ddf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f860c000c00 tx=0x7f860c009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 shutdown_connections 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f8608069370 0x7f860806b820 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f861c0fbe40 0x7f861c10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 --2- 172.21.15.73:0/3425506224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f861c0fcfb0 0x7f861c10ddf0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 >> 172.21.15.73:0/3425506224 conn(0x7f861c0f7500 msgr2=0x7f861c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:49.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 shutdown_connections 2024-02-10T06:41:49.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.534+0000 7f8624734700 1 -- 172.21.15.73:0/3425506224 wait complete. 2024-02-10T06:41:49.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.679+0000 7f4390ccb700 1 -- 172.21.15.73:0/1414929913 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c00a940 con 0x7f438c0fb5e0 2024-02-10T06:41:49.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.680+0000 7f4393f31700 1 -- 172.21.15.73:0/1414929913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 msgr2=0x7f438c0fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.680+0000 7f4393f31700 1 --2- 172.21.15.73:0/1414929913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c0fb9f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f437c004a50 tx=0x7f437c00af40 comp rx=0 tx=0).stop 2024-02-10T06:41:49.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.680+0000 7f4393f31700 1 -- 172.21.15.73:0/1414929913 shutdown_connections 2024-02-10T06:41:49.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.680+0000 7f4393f31700 1 --2- 172.21.15.73:0/1414929913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 0x7f438c0fcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.680+0000 7f4393f31700 1 --2- 172.21.15.73:0/1414929913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c0fb9f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.680+0000 7f4393f31700 1 -- 172.21.15.73:0/1414929913 >> 172.21.15.73:0/1414929913 conn(0x7f438c0f6ca0 msgr2=0x7f438c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:49.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.681+0000 7f4393f31700 1 -- 172.21.15.73:0/1414929913 shutdown_connections 2024-02-10T06:41:49.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.681+0000 7f4393f31700 1 -- 172.21.15.73:0/1414929913 wait complete. 2024-02-10T06:41:49.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.681+0000 7f4393f31700 1 Processor -- start 2024-02-10T06:41:49.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.681+0000 7f4393f31700 1 -- start start 2024-02-10T06:41:49.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4393f31700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c10b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:49.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4393f31700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 0x7f438c10bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:49.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4393f31700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c10c370 con 0x7f438c0fb5e0 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4393f31700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c10cc10 con 0x7f438c0fc7f0 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4391ccd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c10b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4391ccd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c10b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51932/0 (socket says 172.21.15.73:51932) 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f4391ccd700 1 -- 172.21.15.73:0/2614090675 learned_addr learned my addr 172.21.15.73:0/2614090675 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f43914cc700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 0x7f438c10bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f43914cc700 1 -- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 msgr2=0x7f438c10b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f43914cc700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c10b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.682+0000 7f43914cc700 1 -- 172.21.15.73:0/2614090675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c005040 con 0x7f438c0fc7f0 2024-02-10T06:41:49.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.683+0000 7f43914cc700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 0x7f438c10bd70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4388009fd0 tx=0x7f438800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:49.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.683+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4388004e90 con 0x7f438c0fc7f0 2024-02-10T06:41:49.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.683+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c10ce90 con 0x7f438c0fc7f0 2024-02-10T06:41:49.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.683+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c10d3d0 con 0x7f438c0fc7f0 2024-02-10T06:41:49.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.683+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f438800f040 con 0x7f438c0fc7f0 2024-02-10T06:41:49.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.683+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f438800ca50 con 0x7f438c0fc7f0 2024-02-10T06:41:49.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.684+0000 7f4391ccd700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:41:49.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.684+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43700052e0 con 0x7f438c0fc7f0 2024-02-10T06:41:49.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.685+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4388007cc0 con 0x7f438c0fc7f0 2024-02-10T06:41:49.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.686+0000 7f43827fc700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f43780692a0 0x7f437806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:49.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.686+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4388086c70 con 0x7f438c0fc7f0 2024-02-10T06:41:49.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.686+0000 7f4391ccd700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f43780692a0 0x7f437806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:49.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.686+0000 7f4391ccd700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f43780692a0 0x7f437806b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c3b0 tx=0x7f437c00af40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:49.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.693+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f43880569f0 con 0x7f438c0fc7f0 2024-02-10T06:41:49.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:49 smithi073 ceph-mon[29063]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:49.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:49 smithi073 ceph-mon[29063]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:49.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:49 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3425506224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:41:50.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:49.999+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4370005860 con 0x7f438c0fc7f0 2024-02-10T06:41:50.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.000+0000 7f43827fc700 1 -- 172.21.15.73:0/2614090675 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1811 (secure 0 0 0) 0x7f4388019730 con 0x7f438c0fc7f0 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:e14 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:legacy client fscid: 1 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:fs_name cephfs 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 14 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-02-10T06:40:50.835658+0000 2024-02-10T06:41:50.001 INFO:teuthology.orchestra.run.smithi073.stdout:modified 2024-02-10T06:41:08.652904+0000 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:tableserver 0 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:root 0 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:session_timeout 60 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:session_autoclose 300 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:max_file_size 1099511627776 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:required_client_features {} 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure 0 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure_osd_epoch 47 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:max_mds 1 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:in 0 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:up {0=24237} 2024-02-10T06:41:50.002 INFO:teuthology.orchestra.run.smithi073.stdout:failed 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:damaged 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:stopped 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:data_pools [3] 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:metadata_pool 2 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:inline_data enabled 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:balancer 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:standby_count_wanted 1 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi181.cuebex{0:24237} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:Standby daemons: 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi073.ykxzap{-1:14476} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi181.otunfp{-1:24249} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:50.003 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi073.nuvejb{-1:24257} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.73:6826/2661433887,v1:172.21.15.73:6827/2661433887] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.002+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f43780692a0 msgr2=0x7f437806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.002+0000 7f4393f31700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f43780692a0 0x7f437806b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c3b0 tx=0x7f437c00af40 comp rx=0 tx=0).stop 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.002+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 msgr2=0x7f438c10bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.002+0000 7f4393f31700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 0x7f438c10bd70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4388009fd0 tx=0x7f438800b040 comp rx=0 tx=0).stop 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.002+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 shutdown_connections 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.002+0000 7f4393f31700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f43780692a0 0x7f437806b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.003+0000 7f4393f31700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f438c0fb5e0 0x7f438c10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.003+0000 7f4393f31700 1 --2- 172.21.15.73:0/2614090675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f438c0fc7f0 0x7f438c10bd70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.003+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 >> 172.21.15.73:0/2614090675 conn(0x7f438c0f6ca0 msgr2=0x7f438c0f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:50.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.003+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 shutdown_connections 2024-02-10T06:41:50.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.003+0000 7f4393f31700 1 -- 172.21.15.73:0/2614090675 wait complete. 2024-02-10T06:41:50.006 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 14 2024-02-10T06:41:50.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:49 smithi181 ceph-mon[36721]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:50.090 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:49 smithi181 ceph-mon[36721]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:50.091 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:49 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3425506224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.165+0000 7f71d7183700 1 -- 172.21.15.73:0/1708062995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d00708c0 msgr2=0x7f71d0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.165+0000 7f71d7183700 1 --2- 172.21.15.73:0/1708062995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d00708c0 0x7f71d0070d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71bc004940 tx=0x7f71bc009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 -- 172.21.15.73:0/1708062995 shutdown_connections 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 --2- 172.21.15.73:0/1708062995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d00708c0 0x7f71d0070d10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 --2- 172.21.15.73:0/1708062995 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d0071f10 0x7f71d0070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 -- 172.21.15.73:0/1708062995 >> 172.21.15.73:0/1708062995 conn(0x7f71d00f8620 msgr2=0x7f71d00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 -- 172.21.15.73:0/1708062995 shutdown_connections 2024-02-10T06:41:50.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 -- 172.21.15.73:0/1708062995 wait complete. 2024-02-10T06:41:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.166+0000 7f71d7183700 1 Processor -- start 2024-02-10T06:41:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.167+0000 7f71d7183700 1 -- start start 2024-02-10T06:41:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.167+0000 7f71d7183700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d00708c0 0x7f71d010fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:50.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.167+0000 7f71d7183700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 0x7f71d010ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.167+0000 7f71d7183700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d01105b0 con 0x7f71d00708c0 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.167+0000 7f71d7183700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d0110e50 con 0x7f71d0071f10 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71cffff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 0x7f71d010ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71cffff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 0x7f71d010ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.73:57286/0 (socket says 172.21.15.73:57286) 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71cffff700 1 -- 172.21.15.73:0/236943064 learned_addr learned my addr 172.21.15.73:0/236943064 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71cffff700 1 -- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d00708c0 msgr2=0x7f71d010fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.167+0000 7f71d4f1f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d00708c0 0x7f71d010fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:50.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71cffff700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d00708c0 0x7f71d010fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71cffff700 1 -- 172.21.15.73:0/236943064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc005040 con 0x7f71d0071f10 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.168+0000 7f71d4f1f700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d00708c0 0x7f71d010fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.169+0000 7f71cffff700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 0x7f71d010ffb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71bc000c00 tx=0x7f71bc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.169+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71bc00e070 con 0x7f71d0071f10 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.169+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71bc020de0 con 0x7f71d0071f10 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.169+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71bc029c30 con 0x7f71d0071f10 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.169+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d01110d0 con 0x7f71d0071f10 2024-02-10T06:41:50.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.169+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d0111550 con 0x7f71d0071f10 2024-02-10T06:41:50.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.171+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d004bed0 con 0x7f71d0071f10 2024-02-10T06:41:50.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.171+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f71bc0323f0 con 0x7f71d0071f10 2024-02-10T06:41:50.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.172+0000 7f71cd7fa700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71c0069370 0x7f71c006b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:50.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.172+0000 7f71d4f1f700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71c0069370 0x7f71c006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:50.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.172+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f71bc091100 con 0x7f71d0071f10 2024-02-10T06:41:50.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.173+0000 7f71d4f1f700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71c0069370 0x7f71c006b820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71c4004600 tx=0x7f71c400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:50.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.180+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f71bc061e30 con 0x7f71d0071f10 2024-02-10T06:41:50.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.590+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71d0111ae0 con 0x7f71c0069370 2024-02-10T06:41:50.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.591+0000 7f71cd7fa700 1 -- 172.21.15.73:0/236943064 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f71d0111ae0 con 0x7f71c0069370 2024-02-10T06:41:50.592 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-02-10T06:41:50.592 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T06:41:50.592 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-02-10T06:41:50.592 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:41:50.592 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-02-10T06:41:50.593 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "", 2024-02-10T06:41:50.593 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 image", 2024-02-10T06:41:50.593 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-02-10T06:41:50.593 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-02-10T06:41:50.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.594+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71c0069370 msgr2=0x7f71c006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.594+0000 7f71d7183700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71c0069370 0x7f71c006b820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71c4004600 tx=0x7f71c400b040 comp rx=0 tx=0).stop 2024-02-10T06:41:50.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.594+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 msgr2=0x7f71d010ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.594+0000 7f71d7183700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 0x7f71d010ffb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71bc000c00 tx=0x7f71bc009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 shutdown_connections 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f71c0069370 0x7f71c006b820 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f71d00708c0 0x7f71d010fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 --2- 172.21.15.73:0/236943064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0071f10 0x7f71d010ffb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 >> 172.21.15.73:0/236943064 conn(0x7f71d00f8620 msgr2=0x7f71d0102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 shutdown_connections 2024-02-10T06:41:50.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.596+0000 7f71d7183700 1 -- 172.21.15.73:0/236943064 wait complete. 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.727+0000 7f9f97079700 1 -- 172.21.15.73:0/3415808546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fbe50 msgr2=0x7f9f900fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.727+0000 7f9f97079700 1 --2- 172.21.15.73:0/3415808546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fbe50 0x7f9f900fc260 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9f84004940 tx=0x7f9f84009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.727+0000 7f9f97079700 1 -- 172.21.15.73:0/3415808546 shutdown_connections 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.727+0000 7f9f97079700 1 --2- 172.21.15.73:0/3415808546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fcfc0 0x7f9f900fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.727+0000 7f9f97079700 1 --2- 172.21.15.73:0/3415808546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fbe50 0x7f9f900fc260 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.727+0000 7f9f97079700 1 -- 172.21.15.73:0/3415808546 >> 172.21.15.73:0/3415808546 conn(0x7f9f900f7510 msgr2=0x7f9f900f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.728+0000 7f9f97079700 1 -- 172.21.15.73:0/3415808546 shutdown_connections 2024-02-10T06:41:50.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.728+0000 7f9f97079700 1 -- 172.21.15.73:0/3415808546 wait complete. 2024-02-10T06:41:50.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.729+0000 7f9f97079700 1 Processor -- start 2024-02-10T06:41:50.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.729+0000 7f9f97079700 1 -- start start 2024-02-10T06:41:50.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.729+0000 7f9f97079700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fbe50 0x7f9f90113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:50.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.729+0000 7f9f97079700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 0x7f9f90114360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f8ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 0x7f9f90114360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f8ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 0x7f9f90114360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51968/0 (socket says 172.21.15.73:51968) 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f94e15700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fbe50 0x7f9f90113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f94e15700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fbe50 0x7f9f90113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.73:57308/0 (socket says 172.21.15.73:57308) 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f8ffff700 1 -- 172.21.15.73:0/1481896887 learned_addr learned my addr 172.21.15.73:0/1481896887 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90115120 con 0x7f9f900fcfc0 2024-02-10T06:41:50.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90115290 con 0x7f9f900fbe50 2024-02-10T06:41:50.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f8ffff700 1 -- 172.21.15.73:0/1481896887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fbe50 msgr2=0x7f9f90113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:50.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f8ffff700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fbe50 0x7f9f90113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:50.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.730+0000 7f9f8ffff700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f84005040 con 0x7f9f900fcfc0 2024-02-10T06:41:50.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.731+0000 7f9f8ffff700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 0x7f9f90114360 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00b580 tx=0x7f9f7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:50.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.731+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f7c00f6f0 con 0x7f9f900fcfc0 2024-02-10T06:41:50.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.731+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f90115510 con 0x7f9f900fcfc0 2024-02-10T06:41:50.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.731+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f7c00fd20 con 0x7f9f900fcfc0 2024-02-10T06:41:50.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.731+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f7c0103a0 con 0x7f9f900fcfc0 2024-02-10T06:41:50.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.731+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f90115a50 con 0x7f9f900fcfc0 2024-02-10T06:41:50.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.734+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9004bed0 con 0x7f9f900fcfc0 2024-02-10T06:41:50.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.735+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9f7c00f850 con 0x7f9f900fcfc0 2024-02-10T06:41:50.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.735+0000 7f9f8d7fa700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9f800693c0 0x7f9f8006b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:41:50.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.735+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9f7c086dc0 con 0x7f9f900fcfc0 2024-02-10T06:41:50.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.736+0000 7f9f94e15700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9f800693c0 0x7f9f8006b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:41:50.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.737+0000 7f9f94e15700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9f800693c0 0x7f9f8006b870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000c00 tx=0x7f9f84009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:41:50.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:50.744+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9f7c053080 con 0x7f9f900fcfc0 2024-02-10T06:41:51.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:50 smithi073 ceph-mon[29063]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:51.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:50 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/2614090675' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:41:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.126+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9f901038e0 con 0x7f9f900fcfc0 2024-02-10T06:41:51.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.127+0000 7f9f8d7fa700 1 -- 172.21.15.73:0/1481896887 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f9f7c0566a0 con 0x7f9f900fcfc0 2024-02-10T06:41:51.128 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-10T06:41:51.128 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-10T06:41:51.128 INFO:teuthology.orchestra.run.smithi073.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-10T06:41:51.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9f800693c0 msgr2=0x7f9f8006b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:51.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9f800693c0 0x7f9f8006b870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000c00 tx=0x7f9f84009e30 comp rx=0 tx=0).stop 2024-02-10T06:41:51.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 msgr2=0x7f9f90114360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:41:51.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 0x7f9f90114360 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00b580 tx=0x7f9f7c0091a0 comp rx=0 tx=0).stop 2024-02-10T06:41:51.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 shutdown_connections 2024-02-10T06:41:51.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f9f800693c0 0x7f9f8006b870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:51.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f900fbe50 0x7f9f90113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:51.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 --2- 172.21.15.73:0/1481896887 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9f900fcfc0 0x7f9f90114360 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:41:51.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.129+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 >> 172.21.15.73:0/1481896887 conn(0x7f9f900f7510 msgr2=0x7f9f900f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:41:51.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.130+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 shutdown_connections 2024-02-10T06:41:51.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:41:51.130+0000 7f9f97079700 1 -- 172.21.15.73:0/1481896887 wait complete. 2024-02-10T06:41:51.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:50 smithi181 ceph-mon[36721]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:51.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:50 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/2614090675' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:41:52.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:51 smithi073 ceph-mon[29063]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:52.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:51 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/1481896887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:41:52.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:51 smithi181 ceph-mon[36721]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:52.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:51 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/1481896887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:41:53.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:52 smithi073 ceph-mon[29063]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:53.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:52 smithi181 ceph-mon[36721]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:55.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:54 smithi073 ceph-mon[29063]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:55.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:54 smithi181 ceph-mon[36721]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:57.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:56 smithi073 ceph-mon[29063]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:57.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:56 smithi181 ceph-mon[36721]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:58.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:57 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:58.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:57 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:41:59.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:41:58 smithi073 ceph-mon[29063]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:41:59.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:41:58 smithi181 ceph-mon[36721]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:01.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:00 smithi073 ceph-mon[29063]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:01.142 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:00 smithi181 ceph-mon[36721]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:03.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:02 smithi073 ceph-mon[29063]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:03.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:02 smithi181 ceph-mon[36721]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:05.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:04 smithi181 ceph-mon[36721]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:05.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:04 smithi073 ceph-mon[29063]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:07.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:06 smithi181 ceph-mon[36721]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:07.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:06 smithi073 ceph-mon[29063]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:42:09.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:08 smithi181 ceph-mon[36721]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T06:42:09.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:08 smithi073 ceph-mon[29063]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T06:42:11.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:10 smithi181 ceph-mon[36721]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T06:42:11.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:10 smithi073 ceph-mon[29063]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T06:42:13.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:12 smithi181 ceph-mon[36721]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T06:42:13.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:12 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:42:13.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:12 smithi073 ceph-mon[29063]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T06:42:13.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:12 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:42:15.026 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:14 smithi073 ceph-mon[29063]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:42:15.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:14 smithi181 ceph-mon[36721]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:42:15.511 INFO:tasks.workunit.client.1.smithi181.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-10T06:42:15.511 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.511 INFO:tasks.workunit.client.1.smithi181.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T06:42:15.511 INFO:tasks.workunit.client.1.smithi181.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T06:42:15.511 INFO:tasks.workunit.client.1.smithi181.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T06:42:15.511 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: git switch -c 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr:Or undo this operation with: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: git switch - 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr: 2024-02-10T06:42:15.512 INFO:tasks.workunit.client.1.smithi181.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-10T06:42:15.515 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-10T06:42:15.578 INFO:tasks.workunit.client.1.smithi181.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T06:42:15.580 INFO:tasks.workunit.client.1.smithi181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-10T06:42:15.580 INFO:tasks.workunit.client.1.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T06:42:16.405 INFO:tasks.workunit.client.1.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T06:42:16.467 INFO:tasks.workunit.client.1.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T06:42:16.507 INFO:tasks.workunit.client.1.smithi181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-10T06:42:16.509 INFO:tasks.workunit.client.1.smithi181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-10T06:42:16.509 INFO:tasks.workunit.client.1.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T06:42:16.548 INFO:tasks.workunit.client.1.smithi181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-10T06:42:16.551 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:42:16.551 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-10T06:42:16.607 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-10T06:42:16.609 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-10T06:42:16.610 DEBUG:teuthology.orchestra.run.smithi181:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-10T06:42:16.667 INFO:tasks.workunit.client.1.smithi181.stderr:+ mkdir -p fsstress 2024-02-10T06:42:16.668 INFO:tasks.workunit.client.1.smithi181.stderr:+ pushd fsstress 2024-02-10T06:42:16.668 INFO:tasks.workunit.client.1.smithi181.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-10T06:42:16.669 INFO:tasks.workunit.client.1.smithi181.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:42:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:16 smithi073 ceph-mon[29063]: Upgrade: Need to upgrade myself (mgr.smithi073.vjivfn) 2024-02-10T06:42:17.307 INFO:tasks.workunit.client.1.smithi181.stderr:+ tar xzf ltp-full.tgz 2024-02-10T06:42:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-02-10T06:42:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:42:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T06:42:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:42:17.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:17.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' 2024-02-10T06:42:17.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:16 smithi181 ceph-mon[36721]: Upgrade: Need to upgrade myself (mgr.smithi073.vjivfn) 2024-02-10T06:42:18.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:17 smithi073 ceph-mon[29063]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi181 2024-02-10T06:42:18.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:17 smithi181 ceph-mon[36721]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi181 2024-02-10T06:42:19.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:18 smithi073 ceph-mon[29063]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:42:19.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:18 smithi181 ceph-mon[36721]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-10T06:42:21.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:20 smithi073 ceph-mon[29063]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-10T06:42:21.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 -- 172.21.15.73:0/1924321750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fe0e0 msgr2=0x7f60e00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 --2- 172.21.15.73:0/1924321750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fe0e0 0x7f60e00fe550 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60d4004940 tx=0x7f60d4009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 -- 172.21.15.73:0/1924321750 shutdown_connections 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 --2- 172.21.15.73:0/1924321750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fe0e0 0x7f60e00fe550 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 --2- 172.21.15.73:0/1924321750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60e00fcef0 0x7f60e00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 -- 172.21.15.73:0/1924321750 >> 172.21.15.73:0/1924321750 conn(0x7f60e00f8620 msgr2=0x7f60e00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 -- 172.21.15.73:0/1924321750 shutdown_connections 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 -- 172.21.15.73:0/1924321750 wait complete. 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 Processor -- start 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.239+0000 7f60e4eec700 1 -- start start 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60e4eec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 0x7f60e010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60e4eec700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60e00fe0e0 0x7f60e010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60e4eec700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e010e430 con 0x7f60e00fcef0 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60e4eec700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e010ecd0 con 0x7f60e00fe0e0 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 0x7f60e010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 0x7f60e010d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44580/0 (socket says 172.21.15.73:44580) 2024-02-10T06:42:21.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 -- 172.21.15.73:0/820399421 learned_addr learned my addr 172.21.15.73:0/820399421 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:21.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60ddd9b700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60e00fe0e0 0x7f60e010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 -- 172.21.15.73:0/820399421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60e00fe0e0 msgr2=0x7f60e010de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60e00fe0e0 0x7f60e010de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 -- 172.21.15.73:0/820399421 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d4005040 con 0x7f60e00fcef0 2024-02-10T06:42:21.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60de59c700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 0x7f60e010d8f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60d000eb70 tx=0x7f60d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:21.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d0007cc0 con 0x7f60e00fcef0 2024-02-10T06:42:21.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e010ef50 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60d000f040 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d000cb20 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.240+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e010f490 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.242+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e004bed0 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.242+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f60d000cc80 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.242+0000 7f60ceffd700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60c80692a0 0x7f60c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.242+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f60d0086c40 con 0x7f60e00fcef0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.243+0000 7f60ddd9b700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60c80692a0 0x7f60c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.243+0000 7f60ddd9b700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60c80692a0 0x7f60c806b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60d4000c00 tx=0x7f60d4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:21.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.247+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f60d0056930 con 0x7f60e00fcef0 2024-02-10T06:42:21.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:20 smithi181 ceph-mon[36721]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-10T06:42:21.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.451+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60e010f990 con 0x7f60c80692a0 2024-02-10T06:42:21.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.452+0000 7f60ceffd700 1 -- 172.21.15.73:0/820399421 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f60e010f990 con 0x7f60c80692a0 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60c80692a0 msgr2=0x7f60c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60c80692a0 0x7f60c806b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60d4000c00 tx=0x7f60d4009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 msgr2=0x7f60e010d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 0x7f60e010d8f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60d000eb70 tx=0x7f60d000b040 comp rx=0 tx=0).stop 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 shutdown_connections 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f60c80692a0 0x7f60c806b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60e00fcef0 0x7f60e010d8f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 --2- 172.21.15.73:0/820399421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60e00fe0e0 0x7f60e010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 >> 172.21.15.73:0/820399421 conn(0x7f60e00f8620 msgr2=0x7f60e01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:21.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 shutdown_connections 2024-02-10T06:42:21.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.454+0000 7f60e4eec700 1 -- 172.21.15.73:0/820399421 wait complete. 2024-02-10T06:42:21.465 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-02-10T06:42:21.558 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 -- 172.21.15.73:0/2574540592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fcef0 msgr2=0x7f48680fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.558 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 --2- 172.21.15.73:0/2574540592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fcef0 0x7f48680fd320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4860009ce0 tx=0x7f4860004d10 comp rx=0 tx=0).stop 2024-02-10T06:42:21.558 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 -- 172.21.15.73:0/2574540592 shutdown_connections 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 --2- 172.21.15.73:0/2574540592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48680fe0e0 0x7f48680fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 --2- 172.21.15.73:0/2574540592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fcef0 0x7f48680fd320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 -- 172.21.15.73:0/2574540592 >> 172.21.15.73:0/2574540592 conn(0x7f48680f8620 msgr2=0x7f48680faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 -- 172.21.15.73:0/2574540592 shutdown_connections 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.556+0000 7f486fd50700 1 -- 172.21.15.73:0/2574540592 wait complete. 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486fd50700 1 Processor -- start 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486fd50700 1 -- start start 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486fd50700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48680fcef0 0x7f486810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486fd50700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 0x7f486810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486fd50700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486810e4b0 con 0x7f48680fe0e0 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486fd50700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486810ed50 con 0x7f48680fcef0 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486d2eb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 0x7f486810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486d2eb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 0x7f486810deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44606/0 (socket says 172.21.15.73:44606) 2024-02-10T06:42:21.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486daec700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48680fcef0 0x7f486810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486d2eb700 1 -- 172.21.15.73:0/452459863 learned_addr learned my addr 172.21.15.73:0/452459863 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486d2eb700 1 -- 172.21.15.73:0/452459863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48680fcef0 msgr2=0x7f486810d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486d2eb700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48680fcef0 0x7f486810d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.557+0000 7f486d2eb700 1 -- 172.21.15.73:0/452459863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009940 con 0x7f48680fe0e0 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.558+0000 7f486d2eb700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 0x7f486810deb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f485c00eb40 tx=0x7f485c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.558+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f485c007cc0 con 0x7f48680fe0e0 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.558+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f485c00f040 con 0x7f48680fe0e0 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.558+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486810efd0 con 0x7f48680fe0e0 2024-02-10T06:42:21.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.558+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f485c00ca10 con 0x7f48680fe0e0 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.558+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486810f510 con 0x7f48680fe0e0 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.559+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486804bed0 con 0x7f48680fe0e0 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.559+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f485c00cb70 con 0x7f48680fe0e0 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.560+0000 7f48667fc700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48500692a0 0x7f485006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.560+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f485c086d20 con 0x7f48680fe0e0 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.560+0000 7f486daec700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48500692a0 0x7f485006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.560+0000 7f486daec700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48500692a0 0x7f485006b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4860009460 tx=0x7f4860004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:21.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.564+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f485c04dbd0 con 0x7f48680fe0e0 2024-02-10T06:42:21.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.771+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f486810f9b0 con 0x7f48500692a0 2024-02-10T06:42:21.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.772+0000 7f48667fc700 1 -- 172.21.15.73:0/452459863 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f486810f9b0 con 0x7f48500692a0 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48500692a0 msgr2=0x7f485006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48500692a0 0x7f485006b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4860009460 tx=0x7f4860004d10 comp rx=0 tx=0).stop 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 msgr2=0x7f486810deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 0x7f486810deb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f485c00eb40 tx=0x7f485c00b040 comp rx=0 tx=0).stop 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 shutdown_connections 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f48500692a0 0x7f485006b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48680fcef0 0x7f486810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 --2- 172.21.15.73:0/452459863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f48680fe0e0 0x7f486810deb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 >> 172.21.15.73:0/452459863 conn(0x7f48680f8620 msgr2=0x7f48681013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 shutdown_connections 2024-02-10T06:42:21.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.774+0000 7f486fd50700 1 -- 172.21.15.73:0/452459863 wait complete. 2024-02-10T06:42:21.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- 172.21.15.73:0/1016546790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 msgr2=0x7f4c2c102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 --2- 172.21.15.73:0/1016546790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c102520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4c28004940 tx=0x7f4c28009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:21.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- 172.21.15.73:0/1016546790 shutdown_connections 2024-02-10T06:42:21.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 --2- 172.21.15.73:0/1016546790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c102520 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 --2- 172.21.15.73:0/1016546790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c2c0663e0 0x7f4c2c0ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- 172.21.15.73:0/1016546790 >> 172.21.15.73:0/1016546790 conn(0x7f4c2c0f74e0 msgr2=0x7f4c2c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- 172.21.15.73:0/1016546790 shutdown_connections 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- 172.21.15.73:0/1016546790 wait complete. 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 Processor -- start 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- start start 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c2c0663e0 0x7f4c2c10d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c10e4b0 con 0x7f4c2c1000c0 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.878+0000 7f4c339e2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c10ed50 con 0x7f4c2c0663e0 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c10deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44614/0 (socket says 172.21.15.73:44614) 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 -- 172.21.15.73:0/2589440668 learned_addr learned my addr 172.21.15.73:0/2589440668 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c3177e700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c2c0663e0 0x7f4c2c10d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 -- 172.21.15.73:0/2589440668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c2c0663e0 msgr2=0x7f4c2c10d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c2c0663e0 0x7f4c2c10d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 -- 172.21.15.73:0/2589440668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c28005040 con 0x7f4c2c1000c0 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c30f7d700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c10deb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c2800e040 con 0x7f4c2c1000c0 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c10efd0 con 0x7f4c2c1000c0 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c2800bbf0 con 0x7f4c2c1000c0 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c2801c630 con 0x7f4c2c1000c0 2024-02-10T06:42:21.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.879+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c10f510 con 0x7f4c2c1000c0 2024-02-10T06:42:21.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.880+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c04bed0 con 0x7f4c2c1000c0 2024-02-10T06:42:21.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.881+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4c2800bd60 con 0x7f4c2c1000c0 2024-02-10T06:42:21.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.881+0000 7f4c21ffb700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c180692a0 0x7f4c1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:21.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.881+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4c28088560 con 0x7f4c2c1000c0 2024-02-10T06:42:21.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.881+0000 7f4c3177e700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c180692a0 0x7f4c1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:21.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.882+0000 7f4c3177e700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c180692a0 0x7f4c1806b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c004f40 tx=0x7f4c1c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:21.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:21.886+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4c280582d0 con 0x7f4c2c1000c0 2024-02-10T06:42:22.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.083+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c2c10f920 con 0x7f4c180692a0 2024-02-10T06:42:22.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.093+0000 7f4c21ffb700 1 -- 172.21.15.73:0/2589440668 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f4c2c10f920 con 0x7f4c180692a0 2024-02-10T06:42:22.094 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:42:22.094 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (5m) 61s ago 6m 26.7M - 0.25.0 c8568f914cd2 21a01485a256 2024-02-10T06:42:22.094 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (7m) 61s ago 7m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 26dabce1f0be 2024-02-10T06:42:22.094 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi181 smithi181 running (6m) 67s ago 6m 6656k - 17.2.7-505-g72ce7c1d a2a0bd126c96 0ed4bbd7eaf3 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (5m) 61s ago 6m 84.3M - 9.4.7 954c08fa6188 9edb0a6c1f79 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi073.nuvejb smithi073 running (87s) 61s ago 87s 10.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9e00530efbe3 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi073.ykxzap smithi073 running (80s) 61s ago 80s 13.6M - 17.2.7-505-g72ce7c1d a2a0bd126c96 95e7cbe8df86 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi181.cuebex smithi181 running (85s) 67s ago 84s 16.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 e3a8d959df73 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mds.cephfs.smithi181.otunfp smithi181 running (78s) 67s ago 77s 13.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9a844ba6a8fb 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.vjivfn smithi073 *:9283 running (8m) 61s ago 8m 481M - 17.2.7-505-g72ce7c1d a2a0bd126c96 df67d3a17ded 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi181.lmidhc smithi181 *:8443,9283 running (6m) 67s ago 6m 433M - 17.2.7-505-g72ce7c1d a2a0bd126c96 0ede8e2f6c22 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (8m) 61s ago 8m 48.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 9743a7ae51ad 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi181 smithi181 running (5m) 67s ago 5m 35.4M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 335b19e9983e 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (7m) 61s ago 7m 16.2M - 1.5.0 0da6a335fe13 f5df22ebae55 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi181 smithi181 *:9100 running (6m) 67s ago 6m 16.3M - 1.5.0 0da6a335fe13 8b2c66b967b3 2024-02-10T06:42:22.095 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (5m) 61s ago 5m 46.8M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 9218e1a18fa5 2024-02-10T06:42:22.096 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (4m) 61s ago 4m 48.8M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 cffadaeafb44 2024-02-10T06:42:22.096 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (4m) 61s ago 4m 46.7M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 c93fd57a90db 2024-02-10T06:42:22.096 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi181 running (3m) 67s ago 3m 44.0M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 ebadff8f8025 2024-02-10T06:42:22.096 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi181 running (3m) 67s ago 3m 44.5M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 56929cb111f3 2024-02-10T06:42:22.096 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi181 running (3m) 67s ago 3m 41.4M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 85003a026df4 2024-02-10T06:42:22.096 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (5m) 61s ago 6m 52.9M - 2.43.0 a07b618ecd1d 2e631f28238b 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c180692a0 msgr2=0x7f4c1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c180692a0 0x7f4c1806b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c004f40 tx=0x7f4c1c00b040 comp rx=0 tx=0).stop 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 msgr2=0x7f4c2c10deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c10deb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 shutdown_connections 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f4c180692a0 0x7f4c1806b750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c2c0663e0 0x7f4c2c10d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 --2- 172.21.15.73:0/2589440668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4c2c1000c0 0x7f4c2c10deb0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 >> 172.21.15.73:0/2589440668 conn(0x7f4c2c0f74e0 msgr2=0x7f4c2c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:22.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 shutdown_connections 2024-02-10T06:42:22.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.096+0000 7f4c339e2700 1 -- 172.21.15.73:0/2589440668 wait complete. 2024-02-10T06:42:22.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 -- 172.21.15.73:0/2616020615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fe2b0 msgr2=0x7fc8a80fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/2616020615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fe2b0 0x7fc8a80fe700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc89c004940 tx=0x7fc89c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:22.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 -- 172.21.15.73:0/2616020615 shutdown_connections 2024-02-10T06:42:22.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/2616020615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fe2b0 0x7fc8a80fe700 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/2616020615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a80fb220 0x7fc8a80fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 -- 172.21.15.73:0/2616020615 >> 172.21.15.73:0/2616020615 conn(0x7fc8a80f6ca0 msgr2=0x7fc8a80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:22.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 -- 172.21.15.73:0/2616020615 shutdown_connections 2024-02-10T06:42:22.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.198+0000 7fc8acb4b700 1 -- 172.21.15.73:0/2616020615 wait complete. 2024-02-10T06:42:22.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8acb4b700 1 Processor -- start 2024-02-10T06:42:22.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8acb4b700 1 -- start start 2024-02-10T06:42:22.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8acb4b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 0x7fc8a810d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:22.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8acb4b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a80fe2b0 0x7fc8a810ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:22.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8acb4b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a810e4d0 con 0x7fc8a80fb220 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8acb4b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a810ed70 con 0x7fc8a80fe2b0 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8a659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 0x7fc8a810d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8a659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 0x7fc8a810d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44636/0 (socket says 172.21.15.73:44636) 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8a659c700 1 -- 172.21.15.73:0/3004561571 learned_addr learned my addr 172.21.15.73:0/3004561571 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.199+0000 7fc8a5d9b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a80fe2b0 0x7fc8a810ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc8a659c700 1 -- 172.21.15.73:0/3004561571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a80fe2b0 msgr2=0x7fc8a810ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc8a659c700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a80fe2b0 0x7fc8a810ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc8a659c700 1 -- 172.21.15.73:0/3004561571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c005040 con 0x7fc8a80fb220 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc8a659c700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 0x7fc8a810d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc89800eb40 tx=0x7fc89800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc898007cc0 con 0x7fc8a80fb220 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a810eff0 con 0x7fc8a80fb220 2024-02-10T06:42:22.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc89800f040 con 0x7fc8a80fb220 2024-02-10T06:42:22.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.200+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a810f530 con 0x7fc8a80fb220 2024-02-10T06:42:22.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.201+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc89800caa0 con 0x7fc8a80fb220 2024-02-10T06:42:22.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.201+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8880052e0 con 0x7fc8a80fb220 2024-02-10T06:42:22.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.202+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc89800cc00 con 0x7fc8a80fb220 2024-02-10T06:42:22.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.203+0000 7fc896ffd700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc8900692f0 0x7fc89006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:22.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.203+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc898086dd0 con 0x7fc8a80fb220 2024-02-10T06:42:22.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.203+0000 7fc8a5d9b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc8900692f0 0x7fc89006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:22.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.204+0000 7fc8a5d9b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc8900692f0 0x7fc89006b7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009fd0 tx=0x7fc89c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:22.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.215+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc89808a030 con 0x7fc8a80fb220 2024-02-10T06:42:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.776+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc888005d00 con 0x7fc8a80fb220 2024-02-10T06:42:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.778+0000 7fc896ffd700 1 -- 172.21.15.73:0/3004561571 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fc898056a20 con 0x7fc8a80fb220 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 6 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 4 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-02-10T06:42:22.779 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 14 2024-02-10T06:42:22.780 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-02-10T06:42:22.780 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-02-10T06:42:22.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.782+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc8900692f0 msgr2=0x7fc89006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.782+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc8900692f0 0x7fc89006b7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009fd0 tx=0x7fc89c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:22.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.782+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 msgr2=0x7fc8a810d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.782+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 0x7fc8a810d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc89800eb40 tx=0x7fc89800b040 comp rx=0 tx=0).stop 2024-02-10T06:42:22.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.784+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 shutdown_connections 2024-02-10T06:42:22.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.784+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fc8900692f0 0x7fc89006b7a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.784+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8a80fb220 0x7fc8a810d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.784+0000 7fc8acb4b700 1 --2- 172.21.15.73:0/3004561571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a80fe2b0 0x7fc8a810ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.784+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 >> 172.21.15.73:0/3004561571 conn(0x7fc8a80f6ca0 msgr2=0x7fc8a81014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:22.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.786+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 shutdown_connections 2024-02-10T06:42:22.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.786+0000 7fc8acb4b700 1 -- 172.21.15.73:0/3004561571 wait complete. 2024-02-10T06:42:22.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.976+0000 7fb13a7fc700 1 -- 172.21.15.73:0/2156244202 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb12c01c9d0 con 0x7fb13c06e7c0 2024-02-10T06:42:22.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.977+0000 7fb1409a7700 1 -- 172.21.15.73:0/2156244202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c06e7c0 msgr2=0x7fb13c06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.977+0000 7fb1409a7700 1 --2- 172.21.15.73:0/2156244202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c06e7c0 0x7fb13c06ec10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0049c0 tx=0x7fb12c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:22.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.978+0000 7fb1409a7700 1 -- 172.21.15.73:0/2156244202 shutdown_connections 2024-02-10T06:42:22.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.978+0000 7fb1409a7700 1 --2- 172.21.15.73:0/2156244202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c06e7c0 0x7fb13c06ec10 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0049c0 tx=0x7fb12c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:22.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.978+0000 7fb1409a7700 1 --2- 172.21.15.73:0/2156244202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c06e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.978+0000 7fb1409a7700 1 -- 172.21.15.73:0/2156244202 >> 172.21.15.73:0/2156244202 conn(0x7fb13c069ec0 msgr2=0x7fb13c06c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:22.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.979+0000 7fb1409a7700 1 -- 172.21.15.73:0/2156244202 shutdown_connections 2024-02-10T06:42:22.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.979+0000 7fb1409a7700 1 -- 172.21.15.73:0/2156244202 wait complete. 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.980+0000 7fb1409a7700 1 Processor -- start 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.980+0000 7fb1409a7700 1 -- start start 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.980+0000 7fb1409a7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.980+0000 7fb1409a7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c114440 0x7fb13c1152e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.980+0000 7fb1409a7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c114990 con 0x7fb13c114440 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.980+0000 7fb1409a7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c114ad0 con 0x7fb13c06fd80 2024-02-10T06:42:22.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.981+0000 7fb13b7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:22.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.981+0000 7fb13b7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.73:58416/0 (socket says 172.21.15.73:58416) 2024-02-10T06:42:22.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.981+0000 7fb13b7fe700 1 -- 172.21.15.73:0/869532877 learned_addr learned my addr 172.21.15.73:0/869532877 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:22.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.981+0000 7fb13affd700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c114440 0x7fb13c1152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:22.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb13b7fe700 1 -- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c114440 msgr2=0x7fb13c1152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:22.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb13b7fe700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c114440 0x7fb13c1152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:22.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb13b7fe700 1 -- 172.21.15.73:0/869532877 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c005040 con 0x7fb13c06fd80 2024-02-10T06:42:22.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb13b7fe700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c113f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb13000b760 tx=0x7fb1300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:22.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13000f6f0 con 0x7fb13c06fd80 2024-02-10T06:42:22.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c1159c0 con 0x7fb13c06fd80 2024-02-10T06:42:22.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.983+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c115f00 con 0x7fb13c06fd80 2024-02-10T06:42:22.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.984+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb13000fd20 con 0x7fb13c06fd80 2024-02-10T06:42:22.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.985+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13001add0 con 0x7fb13c06fd80 2024-02-10T06:42:22.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.985+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1280052e0 con 0x7fb13c06fd80 2024-02-10T06:42:22.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.986+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb13000f850 con 0x7fb13c06fd80 2024-02-10T06:42:22.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.986+0000 7fb123fff700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb124069160 0x7fb12406b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:22.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.986+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb130086db0 con 0x7fb13c06fd80 2024-02-10T06:42:22.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.988+0000 7fb13affd700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb124069160 0x7fb12406b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:22.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.988+0000 7fb13affd700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb124069160 0x7fb12406b610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb12c000c00 tx=0x7fb12c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:22.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:22.995+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb130056b00 con 0x7fb13c06fd80 2024-02-10T06:42:23.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:22 smithi073 ceph-mon[29063]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:23.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:22 smithi073 ceph-mon[29063]: pgmap v227: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-02-10T06:42:23.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:22 smithi073 ceph-mon[29063]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:23.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:22 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/3004561571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:42:23.199 INFO:tasks.workunit.client.0.smithi073.stderr:Updating files: 79% (9652/12104) Updating files: 80% (9684/12104) Updating files: 81% (9805/12104) Updating files: 82% (9926/12104) Updating files: 83% (10047/12104) Updating files: 84% (10168/12104) Updating files: 85% (10289/12104) Updating files: 86% (10410/12104) Updating files: 87% (10531/12104) Updating files: 88% (10652/12104) Updating files: 89% (10773/12104) Updating files: 90% (10894/12104) Updating files: 91% (11015/12104) Updating files: 92% (11136/12104) Updating files: 93% (11257/12104) Updating files: 94% (11378/12104) Updating files: 95% (11499/12104) Updating files: 96% (11620/12104) Updating files: 97% (11741/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-02-10T06:42:23.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:22 smithi181 ceph-mon[36721]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:23.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:22 smithi181 ceph-mon[36721]: pgmap v227: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-02-10T06:42:23.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:22 smithi181 ceph-mon[36721]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:23.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:22 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/3004561571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:42:23.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.356+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb128005d00 con 0x7fb13c06fd80 2024-02-10T06:42:23.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.357+0000 7fb123fff700 1 -- 172.21.15.73:0/869532877 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1811 (secure 0 0 0) 0x7fb130056690 con 0x7fb13c06fd80 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:e14 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:legacy client fscid: 1 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:fs_name cephfs 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 14 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-02-10T06:40:50.835658+0000 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:modified 2024-02-10T06:41:08.652904+0000 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:tableserver 0 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:root 0 2024-02-10T06:42:23.358 INFO:teuthology.orchestra.run.smithi073.stdout:session_timeout 60 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:session_autoclose 300 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:max_file_size 1099511627776 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:required_client_features {} 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure 0 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure_osd_epoch 47 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:max_mds 1 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:in 0 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:up {0=24237} 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:failed 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:damaged 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:stopped 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:data_pools [3] 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:metadata_pool 2 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:inline_data enabled 2024-02-10T06:42:23.359 INFO:teuthology.orchestra.run.smithi073.stdout:balancer 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout:standby_count_wanted 1 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi181.cuebex{0:24237} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.181:6824/3933807667,v1:172.21.15.181:6825/3933807667] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout:Standby daemons: 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi073.ykxzap{-1:14476} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.73:6828/3906379226,v1:172.21.15.73:6829/3906379226] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi181.otunfp{-1:24249} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.181:6826/1480345027,v1:172.21.15.181:6827/1480345027] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:42:23.360 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.cephfs.smithi073.nuvejb{-1:24257} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.73:6826/2661433887,v1:172.21.15.73:6827/2661433887] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb124069160 msgr2=0x7fb12406b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb124069160 0x7fb12406b610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb12c000c00 tx=0x7fb12c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 msgr2=0x7fb13c113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c113f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb13000b760 tx=0x7fb1300091a0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 shutdown_connections 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fb124069160 0x7fb12406b610 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb13c06fd80 0x7fb13c113f00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 --2- 172.21.15.73:0/869532877 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb13c114440 0x7fb13c1152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.359+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 >> 172.21.15.73:0/869532877 conn(0x7fb13c069ec0 msgr2=0x7fb13c106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:23.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.360+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 shutdown_connections 2024-02-10T06:42:23.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.360+0000 7fb1409a7700 1 -- 172.21.15.73:0/869532877 wait complete. 2024-02-10T06:42:23.362 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 14 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 -- 172.21.15.73:0/3150706316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c0ff150 msgr2=0x7fa36c0ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3150706316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c0ff150 0x7fa36c0ff560 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa35c004940 tx=0x7fa35c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 -- 172.21.15.73:0/3150706316 shutdown_connections 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3150706316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c100360 0x7fa36c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3150706316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c0ff150 0x7fa36c0ff560 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 -- 172.21.15.73:0/3150706316 >> 172.21.15.73:0/3150706316 conn(0x7fa36c0fa920 msgr2=0x7fa36c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 -- 172.21.15.73:0/3150706316 shutdown_connections 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.483+0000 7fa3720b3700 1 -- 172.21.15.73:0/3150706316 wait complete. 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa3720b3700 1 Processor -- start 2024-02-10T06:42:23.485 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa3720b3700 1 -- start start 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa3720b3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c0ff150 0x7fa36c10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa3720b3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c100360 0x7fa36c110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa3720b3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c110680 con 0x7fa36c100360 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa3720b3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c110f20 con 0x7fa36c0ff150 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa36affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c100360 0x7fa36c110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa36affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c100360 0x7fa36c110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44674/0 (socket says 172.21.15.73:44674) 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa36affd700 1 -- 172.21.15.73:0/3670106124 learned_addr learned my addr 172.21.15.73:0/3670106124 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:23.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.484+0000 7fa36b7fe700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c0ff150 0x7fa36c10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:23.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa36b7fe700 1 -- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c100360 msgr2=0x7fa36c110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa36b7fe700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c100360 0x7fa36c110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa36b7fe700 1 -- 172.21.15.73:0/3670106124 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c005040 con 0x7fa36c0ff150 2024-02-10T06:42:23.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa36b7fe700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c0ff150 0x7fa36c10fb40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa35c000c00 tx=0x7fa35c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:23.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35c00e040 con 0x7fa36c0ff150 2024-02-10T06:42:23.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c1111a0 con 0x7fa36c0ff150 2024-02-10T06:42:23.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa35c00bcc0 con 0x7fa36c0ff150 2024-02-10T06:42:23.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35c013660 con 0x7fa36c0ff150 2024-02-10T06:42:23.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.485+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c1116e0 con 0x7fa36c0ff150 2024-02-10T06:42:23.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.486+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c04bed0 con 0x7fa36c0ff150 2024-02-10T06:42:23.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.487+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa35c01c990 con 0x7fa36c0ff150 2024-02-10T06:42:23.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.487+0000 7fa353fff700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3540692a0 0x7fa35406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:23.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.488+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa35c087e80 con 0x7fa36c0ff150 2024-02-10T06:42:23.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.488+0000 7fa36affd700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3540692a0 0x7fa35406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:23.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.488+0000 7fa36affd700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3540692a0 0x7fa35406b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa360004b40 tx=0x7fa360007b40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:23.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.495+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa35c057c40 con 0x7fa36c0ff150 2024-02-10T06:42:23.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.701+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 --> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa36c104cb0 con 0x7fa3540692a0 2024-02-10T06:42:23.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.703+0000 7fa353fff700 1 -- 172.21.15.73:0/3670106124 <== mgr.14212 v2:172.21.15.73:6800/1890404867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fa36c104cb0 con 0x7fa3540692a0 2024-02-10T06:42:23.703 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-02-10T06:42:23.703 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "0/21 daemons upgraded", 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi181", 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-02-10T06:42:23.704 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-02-10T06:42:23.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3540692a0 msgr2=0x7fa35406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3540692a0 0x7fa35406b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa360004b40 tx=0x7fa360007b40 comp rx=0 tx=0).stop 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c0ff150 msgr2=0x7fa36c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c0ff150 0x7fa36c10fb40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa35c000c00 tx=0x7fa35c009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 shutdown_connections 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7fa3540692a0 0x7fa35406b750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.704+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36c0ff150 0x7fa36c10fb40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.705+0000 7fa3720b3700 1 --2- 172.21.15.73:0/3670106124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36c100360 0x7fa36c110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.705+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 >> 172.21.15.73:0/3670106124 conn(0x7fa36c0fa920 msgr2=0x7fa36c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.705+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 shutdown_connections 2024-02-10T06:42:23.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.705+0000 7fa3720b3700 1 -- 172.21.15.73:0/3670106124 wait complete. 2024-02-10T06:42:23.743 INFO:tasks.workunit.client.0.smithi073.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-10T06:42:23.743 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr: git switch -c 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr:Or undo this operation with: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.744 INFO:tasks.workunit.client.0.smithi073.stderr: git switch - 2024-02-10T06:42:23.745 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.745 INFO:tasks.workunit.client.0.smithi073.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T06:42:23.745 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:42:23.745 INFO:tasks.workunit.client.0.smithi073.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-10T06:42:23.748 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-10T06:42:23.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 -- 172.21.15.73:0/1052261031 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fe2b0 msgr2=0x7f47280fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 --2- 172.21.15.73:0/1052261031 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fe2b0 0x7f47280fe700 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4718004940 tx=0x7f4718009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 -- 172.21.15.73:0/1052261031 shutdown_connections 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 --2- 172.21.15.73:0/1052261031 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fe2b0 0x7f47280fe700 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 --2- 172.21.15.73:0/1052261031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47280fb220 0x7f47280fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 -- 172.21.15.73:0/1052261031 >> 172.21.15.73:0/1052261031 conn(0x7f47280f6ca0 msgr2=0x7f47280f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 -- 172.21.15.73:0/1052261031 shutdown_connections 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.808+0000 7f472feb1700 1 -- 172.21.15.73:0/1052261031 wait complete. 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472feb1700 1 Processor -- start 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472feb1700 1 -- start start 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472feb1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 0x7f472810d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472feb1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47280fe2b0 0x7f472810dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472feb1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472810e4c0 con 0x7f47280fb220 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472feb1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472810ed60 con 0x7f47280fe2b0 2024-02-10T06:42:23.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472dc4d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 0x7f472810d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472dc4d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 0x7f472810d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44704/0 (socket says 172.21.15.73:44704) 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472dc4d700 1 -- 172.21.15.73:0/168763857 learned_addr learned my addr 172.21.15.73:0/168763857 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.809+0000 7f472d44c700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47280fe2b0 0x7f472810dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f472dc4d700 1 -- 172.21.15.73:0/168763857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47280fe2b0 msgr2=0x7f472810dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f472dc4d700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47280fe2b0 0x7f472810dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f472dc4d700 1 -- 172.21.15.73:0/168763857 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4718005040 con 0x7f47280fb220 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f472dc4d700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 0x7f472810d980 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f472400eb70 tx=0x7f472400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4724007cc0 con 0x7f47280fb220 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f472400f040 con 0x7f47280fb220 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472810efe0 con 0x7f47280fb220 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f472400cb20 con 0x7f47280fb220 2024-02-10T06:42:23.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.810+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472810f520 con 0x7f47280fb220 2024-02-10T06:42:23.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.811+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472804bed0 con 0x7f47280fb220 2024-02-10T06:42:23.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.811+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f472400cc80 con 0x7f47280fb220 2024-02-10T06:42:23.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.812+0000 7f471e7fc700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f47140692a0 0x7f471406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:42:23.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.812+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4724086be0 con 0x7f47280fb220 2024-02-10T06:42:23.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.812+0000 7f472d44c700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f47140692a0 0x7f471406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:42:23.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.812+0000 7f472d44c700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f47140692a0 0x7f471406b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f471800b0d0 tx=0x7f4718009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:42:23.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:23.816+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f472404dbd0 con 0x7f47280fb220 2024-02-10T06:42:23.897 INFO:tasks.workunit.client.0.smithi073.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T06:42:23.899 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T06:42:23.900 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T06:42:24.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.080+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f472810f900 con 0x7f47280fb220 2024-02-10T06:42:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:23 smithi073 ceph-mon[29063]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:23 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/869532877' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:42:24.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.080+0000 7f471e7fc700 1 -- 172.21.15.73:0/168763857 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f4724056980 con 0x7f47280fb220 2024-02-10T06:42:24.081 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-10T06:42:24.082 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-10T06:42:24.082 INFO:teuthology.orchestra.run.smithi073.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f47140692a0 msgr2=0x7f471406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f47140692a0 0x7f471406b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f471800b0d0 tx=0x7f4718009e30 comp rx=0 tx=0).stop 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 msgr2=0x7f472810d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 0x7f472810d980 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f472400eb70 tx=0x7f472400b040 comp rx=0 tx=0).stop 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 shutdown_connections 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:6800/1890404867,v1:172.21.15.73:6801/1890404867] conn(0x7f47140692a0 0x7f471406b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47280fb220 0x7f472810d980 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 --2- 172.21.15.73:0/168763857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47280fe2b0 0x7f472810dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 >> 172.21.15.73:0/168763857 conn(0x7f47280f6ca0 msgr2=0x7f47281014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:42:24.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 shutdown_connections 2024-02-10T06:42:24.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:42:24.082+0000 7f472feb1700 1 -- 172.21.15.73:0/168763857 wait complete. 2024-02-10T06:42:24.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:23 smithi181 ceph-mon[36721]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:24.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:23 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/869532877' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:42:25.093 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T06:42:25.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:24 smithi073 ceph-mon[29063]: pgmap v228: 65 pgs: 65 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s wr, 40 op/s 2024-02-10T06:42:25.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:24 smithi073 ceph-mon[29063]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:25.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:24 smithi073 ceph-mon[29063]: from='client.? 172.21.15.73:0/168763857' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:42:25.163 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T06:42:25.204 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T06:42:25.206 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T06:42:25.206 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T06:42:25.245 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T06:42:25.248 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:42:25.248 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-10T06:42:25.264 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-10T06:42:25.265 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-10T06:42:25.265 DEBUG:teuthology.orchestra.run.smithi073:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-10T06:42:25.327 INFO:tasks.workunit.client.0.smithi073.stderr:+ mkdir -p fsstress 2024-02-10T06:42:25.331 INFO:tasks.workunit.client.0.smithi073.stderr:+ pushd fsstress 2024-02-10T06:42:25.331 INFO:tasks.workunit.client.0.smithi073.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-10T06:42:25.332 INFO:tasks.workunit.client.0.smithi073.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-10T06:42:25.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:24 smithi181 ceph-mon[36721]: pgmap v228: 65 pgs: 65 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s wr, 40 op/s 2024-02-10T06:42:25.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:24 smithi181 ceph-mon[36721]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:25.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:24 smithi181 ceph-mon[36721]: from='client.? 172.21.15.73:0/168763857' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:42:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:26 smithi073 ceph-mon[29063]: pgmap v229: 65 pgs: 65 active+clean; 26 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 192 op/s 2024-02-10T06:42:27.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:26 smithi181 ceph-mon[36721]: pgmap v229: 65 pgs: 65 active+clean; 26 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 192 op/s 2024-02-10T06:42:28.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:27 smithi073 ceph-mon[29063]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:42:28.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:27 smithi181 ceph-mon[36721]: from='mgr.14212 172.21.15.73:0/4241957374' entity='mgr.smithi073.vjivfn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:42:29.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:28 smithi073 ceph-mon[29063]: pgmap v230: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 449 op/s 2024-02-10T06:42:29.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:28 smithi181 ceph-mon[36721]: pgmap v230: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 449 op/s 2024-02-10T06:42:30.918 INFO:tasks.workunit.client.0.smithi073.stderr:+ tar xzf ltp-full.tgz 2024-02-10T06:42:31.142 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:30 smithi181 ceph-mon[36721]: pgmap v231: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 448 op/s 2024-02-10T06:42:31.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:30 smithi073 ceph-mon[29063]: pgmap v231: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 448 op/s 2024-02-10T06:42:33.173 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:32 smithi073 ceph-mon[29063]: pgmap v232: 65 pgs: 65 active+clean; 67 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.8 MiB/s wr, 459 op/s 2024-02-10T06:42:33.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:32 smithi181 ceph-mon[36721]: pgmap v232: 65 pgs: 65 active+clean; 67 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.8 MiB/s wr, 459 op/s 2024-02-10T06:42:35.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:34 smithi181 ceph-mon[36721]: pgmap v233: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.8 MiB/s wr, 459 op/s 2024-02-10T06:42:35.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:34 smithi073 ceph-mon[29063]: pgmap v233: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.8 MiB/s wr, 459 op/s 2024-02-10T06:42:37.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:36 smithi181 ceph-mon[36721]: pgmap v234: 65 pgs: 65 active+clean; 84 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.1 MiB/s wr, 560 op/s 2024-02-10T06:42:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:36 smithi073 ceph-mon[29063]: pgmap v234: 65 pgs: 65 active+clean; 84 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.1 MiB/s wr, 560 op/s 2024-02-10T06:42:39.339 INFO:tasks.workunit.client.1.smithi181.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-10T06:42:39.340 INFO:tasks.workunit.client.1.smithi181.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-10T06:42:39.340 INFO:tasks.workunit.client.1.smithi181.stderr:+ make 2024-02-10T06:42:39.340 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:38 smithi181 ceph-mon[36721]: pgmap v235: 65 pgs: 65 active+clean; 112 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 7.6 MiB/s wr, 674 op/s 2024-02-10T06:42:39.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:38 smithi073 ceph-mon[29063]: pgmap v235: 65 pgs: 65 active+clean; 112 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 7.6 MiB/s wr, 674 op/s 2024-02-10T06:42:40.645 INFO:tasks.workunit.client.1.smithi181.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-10T06:42:41.341 INFO:journalctl@ceph.mon.smithi181.smithi181.stdout:Feb 10 06:42:40 smithi181 ceph-mon[36721]: pgmap v236: 65 pgs: 65 active+clean; 112 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.6 MiB/s wr, 417 op/s 2024-02-10T06:42:41.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:42:40 smithi073 ceph-mon[29063]: pgmap v236: 65 pgs: 65 active+clean; 112 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.6 MiB/s wr, 417 op/s 2024-02-10T06:43:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:02.400+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:02.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:02.400+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:02.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:02.400+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:03.773 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:03.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:03.773 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:03.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:03.773 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:03.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:03.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:03.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:03.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:04.802 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:04.370+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:04.803 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:04.370+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:04.803 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:04.370+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:04.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:04.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:04.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:05.356+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:05.356+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:05.356+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:05.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:05.281+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:05.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:05.281+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:05.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:05.281+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:05.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:05.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:05.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:06.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:06.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:06.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:06.344+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:06.344+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:06.344+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:06.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:06.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:06.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:07.312+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:07.312+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:07.312+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:07.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:07.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:07.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:07.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:07.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:07.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:07.794+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:07.794+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:07.794+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:08.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:08.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:08.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:08.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:08.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:08.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:08.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:08.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:08.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:08.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:09.602 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:09.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:09.602 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:09.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:09.602 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:09.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:09.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:09.273+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:09.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:09.273+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:09.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:09.273+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:09.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:09.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:09.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:10.365+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:10.365+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:10.365+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:10.247+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:10.247+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:10.247+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:10.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:10.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:10.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:11.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:11.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:11.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:11.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:11.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:11.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:11.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:11.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:11.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:11.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:11.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:11.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:11.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:11.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:12.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:12.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:12.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:12.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:12.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:12.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:12.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:12.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:12.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:12.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:12.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:12.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:13.513 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:13.218+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:13.513 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:13.218+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:13.513 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:13.218+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:13.513 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:13.389+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:13.513 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:13.389+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:13.513 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:13.389+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:13.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:13.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:13.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:14.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:14.200+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:14.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:14.200+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:14.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:14.200+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:14.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:14.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:14.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:14.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:14.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:14.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:14.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:14.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:14.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:15.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:15.219+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:15.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:15.219+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:15.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:15.219+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:15.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:15.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:15.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:15.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:15.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:15.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:15.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:15.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:16.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:16.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:16.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:16.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:16.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:16.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:16.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:16.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:16.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:16.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:16.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:16.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:16.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:16.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:17.485 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:17.167+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:17.485 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:17.167+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:17.485 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:17.167+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:17.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:17.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:17.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:17.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:17.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:17.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:17.843+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:17.843+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:17.843+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:18.456 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:18.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:18.456 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:18.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:18.456 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:18.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:18.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:18.454+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:18.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:18.454+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:18.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:18.454+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:18.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:18.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:18.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:19.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:19.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:19.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:19.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:19.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:19.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:19.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:19.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:19.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:19.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:19.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:19.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:19.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:19.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:19.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:20.476 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:20.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:20.476 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:20.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:20.477 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:20.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:20.837 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:20.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:20.837 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:20.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:20.837 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:20.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:20.836+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:20.836+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:20.836+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:21.484 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:21.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:21.484 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:21.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:21.484 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:21.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:21.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:21.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:21.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:21.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:21.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:21.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:22.228+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:22.228+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:22.228+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:22.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:22.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:22.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:22.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:22.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:22.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:23.513 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:23.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:23.514 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:23.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:23.514 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:23.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:23.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:23.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:23.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:23.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:23.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:23.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:23.866+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:23.866+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:23.866+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:24.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:24.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:24.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:24.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:24.186+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:24.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:24.186+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:24.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:24.186+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:25.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:24.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:25.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:24.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:25.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:24.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:25.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:25.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:25.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:25.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:25.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:25.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:26.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:26.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:26.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:25.936+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:25.936+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:25.936+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:26.660 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:26.380+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:26.660 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:26.380+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:26.660 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:26.380+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:26.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:26.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:26.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:27.377+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:27.377+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:27.377+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:27.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:27.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:27.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:27.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:27.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:27.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:28.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:27.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:28.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:27.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:28.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:27.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:28.348 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:28.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:28.348 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:28.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:28.348 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:28.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:28.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:28.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:28.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:29.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:28.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:29.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:28.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:29.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:28.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:29.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:29.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:29.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:29.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:29.176+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:29.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:29.176+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:29.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:29.176+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:30.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:30.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:30.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:30.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:30.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:30.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:29.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:29.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:29.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:30.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:30.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:30.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:31.181 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:30.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:31.181 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:30.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:31.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:30.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:31.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:31.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:31.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:31.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:31.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:31.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:31.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:31.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:31.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:32.164 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:31.912+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:32.164 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:31.912+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:32.164 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:31.912+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:32.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:32.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:32.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:32.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:32.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:32.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:32.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:32.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:32.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:32.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:33.395 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:33.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:33.395 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:33.207+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:33.395 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:33.207+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:33.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:33.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:33.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:33.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:33.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:33.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:33.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:33.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:33.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:34.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:34.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:34.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:34.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:34.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:34.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:34.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:34.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:34.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:35.442 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:35.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:35.443 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:35.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:35.443 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:35.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:35.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:35.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:35.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:36.162 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:35.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:36.162 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:35.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:36.162 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:35.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:36.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:36.161+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:36.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:36.161+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:36.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:36.161+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:36.773 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:36.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:36.773 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:36.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:36.773 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:36.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:36.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:36.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:36.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:37.497 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:37.142+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:37.497 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:37.142+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:37.497 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:37.142+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:37.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:37.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:37.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:37.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:37.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:37.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:38.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:38.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:38.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:37.866+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:37.866+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:37.866+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:38.533+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:38.533+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:38.533+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:39.127+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:39.127+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:39.127+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:38.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:38.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:38.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:39.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:39.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:39.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:40.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:40.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:40.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:39.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:39.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:39.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:40.849 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:40.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:40.850 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:40.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:40.850 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:40.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:40.849+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:40.849+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:40.849+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:41.547 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:41.136+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:41.547 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:41.136+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:41.547 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:41.136+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:41.546+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:41.546+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:41.546+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:42.156 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:41.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:42.156 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:41.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:42.156 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:41.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:42.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:42.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:42.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:42.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:42.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:42.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:42.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:42.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:42.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:43.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:42.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:43.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:42.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:43.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:42.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:43.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:43.177+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:43.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:43.177+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:43.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:43.177+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:43.611+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:43.611+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:43.611+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:44.227 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:43.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:44.227 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:43.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:44.227 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:43.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:44.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:44.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:44.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:44.951 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:44.657+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:44.951 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:44.657+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:44.951 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:44.657+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:45.198+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:45.198+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:45.198+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:44.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:44.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:44.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:45.975 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:45.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:45.975 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:45.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:45.975 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:45.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:46.191+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:46.191+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:46.191+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:45.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:45.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:45.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:46.977 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:46.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:46.977 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:46.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:46.977 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:46.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:46.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:46.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:46.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:47.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:47.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:47.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:48.012 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:47.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:48.013 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:47.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:48.013 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:47.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:48.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:48.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:48.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:48.193+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:48.193+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:48.193+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:48.992 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:48.680+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:48.992 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:48.680+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:48.993 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:48.680+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:48.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:48.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:48.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:49.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:49.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:49.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:50.041 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:49.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:50.041 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:49.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:50.041 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:49.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:50.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:50.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:50.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:50.242+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:50.242+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:50.242+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:51.035 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:50.709+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:51.035 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:50.709+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:51.036 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:50.709+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:51.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:51.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:51.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:51.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:51.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:51.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:52.071 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:51.751+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:52.071 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:51.751+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:52.071 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:51.751+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:52.158+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:52.158+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:52.158+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:52.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:52.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:52.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:53.023 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:52.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:53.023 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:52.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:53.023 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:52.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:53.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:53.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:53.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:53.188+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:53.188+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:53.188+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:53.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:53.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:53.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:54.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:54.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:54.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:54.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:54.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:54.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:55.031 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:54.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:55.031 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:54.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:55.031 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:54.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:55.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:55.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:55.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:55.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:55.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:55.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:55.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:55.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:55.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:56.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:56.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:56.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:56.171+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:56.171+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:56.171+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:57.003 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:56.814+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:57.003 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:56.814+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:57.003 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:56.814+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:57.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:57.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:57.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:57.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:57.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:57.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:58.098 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:57.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:58.099 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:57.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:58.099 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:57.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:58.099 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:57.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:58.099 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:57.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:58.099 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:57.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:58.097+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:58.097+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:58.097+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:59.081 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:58.797+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:59.081 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:58.797+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:59.081 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:58.797+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:43:59.081 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:58.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:59.081 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:58.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:59.081 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:58.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:43:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:59.079+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:59.079+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:43:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:43:59.079+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:59.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:59.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:43:59.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:59.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:59.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:43:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:43:59.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:00.123+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:00.123+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:00.123+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:01.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:00.778+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:01.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:00.778+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:01.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:00.778+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:01.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:01.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:01.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:01.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:01.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:01.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:01.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:01.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:01.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:02.048 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:01.789+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:02.048 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:01.789+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:02.048 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:01.789+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:02.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:02.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:02.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:02.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:02.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:02.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:03.086 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:02.821+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:03.086 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:02.821+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:03.086 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:02.821+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:03.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:03.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:03.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:03.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:03.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:03.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:03.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:03.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:03.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:04.064 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:03.791+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:04.065 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:03.791+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:04.065 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:03.791+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:04.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:04.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:04.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:04.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:04.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:04.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:05.108 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:04.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:05.108 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:04.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:05.108 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:04.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:05.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:05.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:05.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:05.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:05.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:05.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:05.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:05.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:05.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:06.086 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:05.822+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:06.087 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:05.822+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:06.087 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:05.822+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:06.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:06.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:06.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:06.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:06.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:06.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:06.086+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:06.086+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:06.086+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:07.021 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:06.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:07.021 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:06.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:07.021 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:06.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:07.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:07.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:07.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:07.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:07.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:07.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:07.773+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:07.773+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:07.773+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:07.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:07.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:07.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:08.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:08.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:08.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:09.108 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:08.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:09.109 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:08.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:09.109 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:08.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:09.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:08.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:09.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:08.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:09.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:08.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:09.108+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:09.108+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:09.108+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:10.079 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:09.724+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:10.079 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:09.724+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:10.079 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:09.724+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:10.079 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:09.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:10.079 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:09.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:10.079 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:09.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:10.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:10.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:10.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:11.036 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:10.743+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:11.037 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:10.743+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:11.037 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:10.743+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:11.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:10.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:11.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:10.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:11.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:10.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:11.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:11.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:11.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:11.938 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:11.747+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:11.939 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:11.747+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:11.939 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:11.747+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:12.193 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:11.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:12.193 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:11.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:12.193 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:11.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:12.193 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:11.999+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:12.193 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:11.999+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:12.193 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:11.999+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:12.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:12.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:12.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:12.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:12.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:12.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:12.971 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:12.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:12.972 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:12.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:12.972 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:12.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:12.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:12.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:12.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:13.671+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:13.671+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:13.671+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:14.018+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:14.018+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:14.018+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:13.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:13.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:13.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:14.977 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:14.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:14.977 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:14.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:14.977 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:14.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:15.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:14.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:15.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:14.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:15.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:14.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:15.266 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:14.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:15.266 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:14.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:15.267 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:14.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:15.591+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:15.591+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:15.591+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:15.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:15.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:15.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:15.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:15.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:15.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:16.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:16.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:16.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:17.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:17.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:17.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:16.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:16.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:16.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:17.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:17.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:17.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:18.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:17.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:18.338 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:17.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:18.338 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:17.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:18.338 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:17.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:18.338 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:17.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:18.338 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:17.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:18.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:18.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:18.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:18.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:18.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:18.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:19.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:19.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:19.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:19.959 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:19.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:19.959 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:19.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:19.959 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:19.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:20.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:19.958+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:20.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:19.958+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:19.958+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:20.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:20.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:20.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:20.922 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:20.668+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:20.923 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:20.668+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:20.923 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:20.668+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:20.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:20.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:20.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:21.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:21.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:21.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:21.905 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:21.689+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:21.906 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:21.689+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:21.906 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:21.689+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:21.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:21.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:21.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:22.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:22.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:22.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:23.044 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:22.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:23.045 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:22.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:23.045 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:22.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:23.045 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:22.723+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:23.045 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:22.723+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:23.045 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:22.723+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:23.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:23.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:23.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:23.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:23.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:23.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:23.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:23.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:23.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:24.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:24.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:24.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:24.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:24.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:24.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:24.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:24.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:24.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:25.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:25.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:25.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:25.935 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:25.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:25.935 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:25.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:25.935 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:25.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:26.100+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:26.100+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:26.100+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:25.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:25.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:25.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:26.981 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:26.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:26.981 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:26.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:26.981 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:26.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:26.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:26.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:26.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:27.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:27.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:27.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:27.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:27.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:27.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:27.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:28.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:28.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:28.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:28.126+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:28.126+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:28.126+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:29.043 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:28.662+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:29.044 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:28.662+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:29.044 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:28.662+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:29.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:29.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:29.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:29.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:29.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:29.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:29.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:29.996 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:29.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:29.996 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:29.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:29.997 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:29.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:30.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:29.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:30.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:29.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:30.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:29.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:30.117+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:30.117+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:30.117+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:30.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:30.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:30.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:31.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:30.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:31.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:30.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:31.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:30.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:31.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:31.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:31.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:31.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:31.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:31.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:32.019+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:32.019+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:32.019+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:32.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:32.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:32.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:32.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:32.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:32.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:33.026+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:33.026+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:33.026+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:33.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:33.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:33.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:33.697 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:33.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:33.697 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:33.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:33.697 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:33.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:34.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:34.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:34.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:34.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:34.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:34.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:34.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:34.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:34.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:34.536+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:34.536+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:34.536+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:35.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:35.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:35.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:35.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:35.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:35.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:35.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:35.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:35.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:36.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:36.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:36.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:36.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:36.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:36.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:36.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:36.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:36.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:36.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:36.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:36.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:37.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:37.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:37.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:37.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:37.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:37.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:37.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:37.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:37.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:37.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:37.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:37.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:38.133+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:38.133+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:38.133+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:38.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:38.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:38.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:38.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:38.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:38.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:39.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:39.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:39.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:39.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:39.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:39.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:39.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:39.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:39.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:40.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:40.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:40.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:40.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:40.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:40.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:40.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:40.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:40.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:41.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:41.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:41.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:41.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:41.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:41.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:41.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:41.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:41.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:41.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:41.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:41.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:42.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:42.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:42.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:42.103+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:42.103+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:42.103+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:42.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:42.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:42.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:43.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:43.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:43.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:43.119+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:43.119+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:43.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:43.119+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:43.501+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:43.501+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:43.501+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:44.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:44.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:44.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:44.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:44.157+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:44.157+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:44.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:44.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:44.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:45.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:45.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:45.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:45.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:45.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:45.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:45.506+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:45.506+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:45.506+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:46.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:46.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:46.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:46.201+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:46.201+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:46.201+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:46.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:44:46.656+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:44:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:46.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:46.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:46.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:46.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:46.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:47.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:47.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:47.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:47.151+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:47.151+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:47.151+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:47.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:47.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:47.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:48.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:48.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:48.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:48.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:48.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:48.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:48.524+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:48.524+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:48.524+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:49.056+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:49.056+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:49.056+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:49.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:49.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:49.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:49.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:49.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:49.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:50.096+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:50.096+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:50.096+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:50.103+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:50.103+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:50.103+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:50.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:50.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:50.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:51.073+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:51.073+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:51.073+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:51.060+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:51.060+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:51.060+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:51.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:44:51.657+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:44:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:51.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:51.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:51.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:52.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:52.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:52.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:52.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:52.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:52.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:52.565+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:52.565+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:52.565+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:53.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:53.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:53.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:53.060+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:53.060+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:53.060+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:53.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:53.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:53.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:54.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:54.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:54.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:54.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:54.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:54.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:54.554+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:54.554+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:54.554+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:55.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:55.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:55.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:54.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:54.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:54.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:55.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:55.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:55.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:56.096+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:56.096+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:56.096+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:55.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:55.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:55.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:56.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:44:56.658+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:44:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:56.617+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:56.617+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:56.617+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:57.002+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:57.002+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:57.002+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:57.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:57.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:57.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:57.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:57.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:57.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:58.039+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:58.039+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:58.039+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:58.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:58.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:58.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:58.580+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:58.580+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:58.580+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:59.075+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:59.075+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:44:59.075+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:44:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:59.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:59.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:44:59.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:44:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:59.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:59.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:44:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:44:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:44:59.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:00.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:00.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:00.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:00.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:00.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:00.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:00.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:00.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:00.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:01.346 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:01.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:01.346 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:01.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:01.346 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:01.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:01.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:01.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:01.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:01.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:01.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:01.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:01.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:01.658+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 31 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:01.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:01.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:01.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:01.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:01.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:01.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:02.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:02.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:02.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:02.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:02.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:02.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:02.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:02.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:02.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:03.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:03.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:03.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:03.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:03.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:03.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:03.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:03.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:03.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:03.986+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:03.986+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:03.986+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:04.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:04.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:04.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:04.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:04.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:04.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:04.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:04.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:04.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:05.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:05.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:05.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:05.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:05.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:05.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:05.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:05.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:05.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:06.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:06.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:06.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:06.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:06.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:06.659+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 66 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:06.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:06.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:06.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:07.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:07.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:07.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:06.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:06.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:06.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:07.550+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:07.550+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:07.550+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:08.094+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:08.094+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:08.094+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:07.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:07.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:07.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:08.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:08.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:08.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:09.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:09.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:09.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:08.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:08.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:08.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:09.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:09.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:09.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:10.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:10.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:10.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:09.902+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:09.902+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:09.902+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:10.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:10.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:10.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:11.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:11.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:11.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:10.915+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:10.915+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:10.915+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:11.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:11.659+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 93 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:11.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:11.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:11.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:11.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:11.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:11.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:12.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:12.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:12.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:11.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:11.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:11.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:12.918 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:12.628+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:12.918 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:12.628+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:12.919 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:12.628+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:13.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:13.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:13.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:12.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:12.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:12.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:13.668+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:13.668+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:13.668+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:13.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:13.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:13.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:14.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:14.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:14.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:15.109 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:14.687+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:15.109 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:14.687+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:15.109 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:14.687+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:15.109 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:14.869+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:15.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:14.869+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:15.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:14.869+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:15.108+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:15.108+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:15.108+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:16.077 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:15.737+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:16.077 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:15.737+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:16.077 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:15.737+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:16.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:15.870+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:16.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:15.870+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:16.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:15.870+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:16.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:16.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:16.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:16.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:16.660+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 124 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:16.921 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:16.692+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:16.921 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:16.692+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:16.921 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:16.692+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:16.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:16.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:16.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:17.035+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:17.035+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:17.035+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:17.996 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:17.669+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:17.996 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:17.669+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:17.996 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:17.669+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:17.997 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:17.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:17.997 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:17.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:17.997 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:17.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:17.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:17.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:17.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:18.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:18.625+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:18.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:18.625+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:18.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:18.625+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:18.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:18.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:18.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:19.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:19.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:19.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:19.583+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:19.583+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:19.583+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:19.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:19.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:19.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:20.024+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:20.024+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:20.024+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:20.894 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:20.626+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:20.894 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:20.626+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:20.894 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:20.626+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:20.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:20.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:20.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:21.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:21.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:21.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:21.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:21.660+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 151 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:21.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:21.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:21.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:21.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:21.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:21.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:21.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:21.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:21.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:22.849 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:22.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:22.849 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:22.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:22.849 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:22.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:22.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:22.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:22.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:22.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:22.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:22.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:23.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:23.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:23.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:23.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:23.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:23.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:23.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:23.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:23.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:24.547+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:24.547+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:24.547+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:24.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:24.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:24.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:24.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:24.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:24.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:25.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:25.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:25.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:25.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:25.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:25.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:25.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:25.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:25.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:26.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:26.661+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 180 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:26.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:26.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:26.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:26.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:26.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:26.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:26.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:26.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:26.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:26.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:26.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:26.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:27.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:27.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:27.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:27.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:27.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:27.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:27.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:27.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:27.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:27.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:28.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:27.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:28.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:27.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:28.821 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:28.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:28.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:28.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:28.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:28.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:28.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:28.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:28.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:28.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:28.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:28.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:29.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:29.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:29.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:29.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:29.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:29.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:29.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:29.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:29.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:29.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:29.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:29.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:30.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:30.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:30.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:30.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:30.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:30.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:30.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:30.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:30.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:30.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:30.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:30.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:31.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:31.662+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 207 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:31.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:31.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:31.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:31.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:31.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:31.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:32.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:31.938+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:32.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:31.938+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:32.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:31.938+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:32.065 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:31.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:32.065 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:31.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:32.065 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:31.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:32.805 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:32.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:32.805 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:32.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:32.805 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:32.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:32.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:32.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:32.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:32.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:32.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:32.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:33.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:33.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:33.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:33.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:33.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:33.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:33.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:33.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:33.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:33.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:33.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:33.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:34.764 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:34.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:34.765 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:34.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:34.765 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:34.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:35.046 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:34.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:35.047 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:34.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:35.047 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:34.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:35.046+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:35.046+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:35.046+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:35.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:35.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:35.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:35.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:35.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:35.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:36.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:35.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:36.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:35.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:36.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:35.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:36.090+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:36.090+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:36.090+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:36.786 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:36.662+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 234 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:36.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:36.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:36.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:36.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:36.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:36.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:37.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:36.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:37.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:36.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:37.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:36.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:37.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:37.082+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:37.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:37.082+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:37.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:37.082+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:37.450+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:37.450+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:37.450+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:37.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:37.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:37.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:38.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:38.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:38.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:38.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:38.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:38.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:38.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:38.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:38.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:39.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:39.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:39.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:39.519+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:39.519+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:39.519+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:39.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:39.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:39.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:40.013+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:40.013+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:40.013+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:40.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:40.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:40.483+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:40.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:40.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:40.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:40.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:40.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:40.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:41.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:41.663+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 261 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:41.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:41.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:41.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:41.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:41.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:41.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:41.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:41.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:41.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:41.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:41.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:41.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:42.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:42.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:42.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:42.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:42.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:42.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:42.941+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:42.941+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:42.941+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:43.449+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:43.449+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:43.449+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:44.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:43.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:44.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:43.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:44.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:43.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:44.353 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:43.986+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:44.353 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:43.986+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:44.354 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:43.986+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:44.433+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:44.433+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:44.433+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:44.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:44.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:44.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:45.009+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:45.009+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:45.009+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:45.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:45.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:45.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:45.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:45.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:45.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:45.891+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:45.891+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:45.891+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:46.674 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:46.664+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 294 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:46.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:46.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:46.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:46.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:46.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:46.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:46.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:46.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:46.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:47.023+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:47.023+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:47.023+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:47.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:47.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:47.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:47.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:47.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:47.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:48.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:48.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:48.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:48.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:48.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:48.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:48.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:48.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:48.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:48.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:48.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:48.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:49.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:49.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:49.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:49.953+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:49.953+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:49.953+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:49.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:49.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:49.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:50.406+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:50.406+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:50.406+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:50.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:50.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:50.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:50.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:50.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:50.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:51.677 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:51.664+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 321 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:51.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:51.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:51.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:51.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:51.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:51.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:51.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:51.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:51.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:52.408+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:52.408+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:52.408+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:52.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:52.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:52.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:52.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:52.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:52.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:53.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:53.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:53.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:54.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:54.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:54.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:53.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:53.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:53.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:54.470+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:54.470+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:54.470+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:54.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:54.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:54.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:55.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:55.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:55.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:55.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:55.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:55.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:55.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:55.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:55.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:56.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:56.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:56.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:56.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:56.666 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:56.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:56.666 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:56.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:56.666 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:56.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:56.992 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:45:56.665+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 348 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:45:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:57.144+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:57.144+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:57.144+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:56.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:56.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:56.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:57.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:57.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:57.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:58.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:58.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:58.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:57.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:57.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:57.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:58.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:58.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:58.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:59.167+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:59.167+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:45:59.167+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:45:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:58.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:58.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:58.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:45:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:59.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:59.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:45:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:45:59.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:00.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:00.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:00.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:59.982+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:59.982+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:45:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:45:59.982+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:00.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:00.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:00.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:01.147+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:01.147+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:01.147+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:00.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:00.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:00.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:01.790 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:01.665+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 375 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:01.791 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:01.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:01.791 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:01.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:01.791 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:01.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:01.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:01.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:01.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:02.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:02.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:02.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:02.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:02.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:02.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:02.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:02.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:02.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:02.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:02.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:02.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:03.442 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:03.218+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:03.442 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:03.218+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:03.442 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:03.218+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:03.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:03.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:03.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:04.237 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:03.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:04.238 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:03.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:04.238 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:03.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:04.237+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:04.237+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:04.237+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:04.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:04.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:04.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:05.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:04.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:05.211 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:04.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:05.211 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:04.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:05.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:05.210+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:05.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:05.210+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:05.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:05.210+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:05.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:05.526+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:05.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:05.526+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:05.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:05.526+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:05.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:05.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:05.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:06.571 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:06.245+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:06.571 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:06.245+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:06.571 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:06.245+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:06.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:06.666+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 410 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:06.570+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:06.570+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:06.570+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:07.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:06.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:07.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:06.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:07.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:06.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:07.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:07.209+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:07.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:07.209+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:07.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:07.209+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:07.522 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:07.209+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:07.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:07.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:07.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:07.521+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:07.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:08.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:07.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:08.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:07.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:08.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:07.912+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.560 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:08.250+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:08.561 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:08.250+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:08.561 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:08.250+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:08.561 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:08.250+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:08.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:08.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:08.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:08.560+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:09.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:08.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:09.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:08.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:09.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:08.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:09.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:08.940+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:09.561 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:09.229+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:09.561 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:09.229+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:09.561 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:09.229+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:09.562 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:09.229+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:09.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:09.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:09.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:09.561+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:10.197 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:09.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:10.197 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:09.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:10.197 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:09.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:10.197 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:09.912+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:10.579 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:10.196+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:10.579 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:10.196+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:10.580 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:10.196+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:10.580 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:10.196+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:10.578+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:10.578+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:10.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:10.578+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:10.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:10.579+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:11.158+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:11.158+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:11.158+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:11.158+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:10.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:10.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:10.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:10.939+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:11.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:11.666+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 437 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:11.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:11.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:11.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:11.614+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:12.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:12.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:12.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:12.115+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:11.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:11.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:11.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:11.962+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:12.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:12.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:12.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:12.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:12.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:12.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:12.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:12.623+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:12.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:12.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:12.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:12.915+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:13.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:13.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:13.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:13.156+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:13.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:13.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:13.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:13.575+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:14.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:13.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:14.187 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:13.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:14.187 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:13.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:14.187 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:13.919+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:14.598 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:14.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:14.598 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:14.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:14.598 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:14.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:14.598 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:14.186+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:14.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:14.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:14.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:14.597+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:14.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:14.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:14.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:14.909+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:15.148+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:15.148+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:15.148+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:15.148+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:15.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:15.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:15.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:15.575+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:16.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:16.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:16.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:16.109+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:15.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:15.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:15.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:15.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:16.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:16.667+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 468 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:16.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:16.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:16.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:16.584+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:17.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:17.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:17.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:17.115+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:16.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:16.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:16.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:16.899+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:17.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:17.607+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:17.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:17.607+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:17.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:17.607+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:17.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:17.607+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:18.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:17.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:18.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:17.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:18.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:17.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:18.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:17.850+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:18.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:18.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:18.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:18.116+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:19.088 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:18.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:19.088 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:18.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:19.088 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:18.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:19.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:18.808+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:19.089 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:18.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:19.089 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:18.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:19.089 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:18.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:19.089 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:18.655+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:19.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:19.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:19.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:19.087+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.071 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:19.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:19.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:19.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:19.840+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:19.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:19.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:19.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:20.072 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:19.653+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:20.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:20.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:20.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:20.071+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:20.608+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:20.608+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:20.608+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:20.608+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:20.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:20.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:20.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:20.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:20.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:20.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:20.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:21.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:21.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:21.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:21.084+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:21.668+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 498 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:22.112 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:21.651+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:22.112 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:21.651+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:22.112 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:21.651+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:22.113 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:21.651+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:21.814+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:22.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:21.814+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:22.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:21.814+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:22.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:21.815+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:22.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:22.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:22.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:22.111+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:22.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:22.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:22.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:22.768+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:22.634+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:22.634+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:22.634+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:22.634+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:23.123+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:23.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:23.123+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:23.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:23.123+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:23.527 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:23.123+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:23.604+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:23.810 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:23.604+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:23.810 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:23.604+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:23.810 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:23.604+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:24.092 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:23.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:24.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:23.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:24.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:23.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:24.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:23.808+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:24.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:24.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:24.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:24.091+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:24.575+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:24.575+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:24.575+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:24.575+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:24.814+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:24.814+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:24.814+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:24.814+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:25.101+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:25.101+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:25.101+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:25.101+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:25.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:25.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:25.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:25.601+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:25.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:25.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:25.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:25.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:25.787+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:26.114+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:26.114+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:26.114+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:26.114+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:26.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:26.669+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 527 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:26.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:26.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:26.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:26.597+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:26.822+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:26.822+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:26.822+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:26.822+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:27.377 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:27.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:27.378 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:27.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:27.378 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:27.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:27.378 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:27.124+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.102 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:27.643+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:28.102 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:27.643+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:28.102 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:27.643+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:28.102 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:27.643+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:27.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:28.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:27.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:28.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:27.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:28.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:27.819+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:28.101+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:28.101+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:28.101+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:28.101+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:29.100 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:28.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:29.100 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:28.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:29.100 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:28.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:29.101 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:28.667+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:29.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:28.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:29.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:28.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:29.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:28.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:29.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:28.784+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:29.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:29.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:29.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:29.099+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:30.093 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:29.776+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:30.093 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:29.776+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:30.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:29.776+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:30.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:29.776+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:30.094 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:29.692+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:30.094 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:29.692+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:30.094 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:29.692+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:30.094 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:29.692+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:30.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:30.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:30.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:30.093+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:30.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:30.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:30.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:30.759+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:30.669+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:30.669+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:31.053 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:30.669+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:31.054 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:30.669+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:31.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:31.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:31.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:31.052+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:32.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:31.669+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 554 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:32.058 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:31.761+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:32.058 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:31.761+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:32.059 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:31.761+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:32.059 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:31.761+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:32.059 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:31.703+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:32.059 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:31.703+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:32.059 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:31.703+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:32.059 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:31.703+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:32.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:32.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:32.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:32.057+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.066 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:32.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:33.066 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:32.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:33.066 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:32.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:33.066 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:32.790+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.067 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:32.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:33.067 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:32.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:33.067 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:32.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:33.067 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:32.718+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:33.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:33.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:33.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:33.065+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:33.673+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:33.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:33.673+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:33.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:33.673+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:33.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:33.673+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:34.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:33.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:34.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:33.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:34.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:33.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:34.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:33.787+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:34.074+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:34.074+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:34.074+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:34.074+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:35.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:34.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:35.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:34.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:35.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:34.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:35.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:34.816+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:35.116 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:34.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:35.116 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:34.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:35.117 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:34.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:35.117 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:34.684+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:35.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:35.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:35.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:35.116+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.114 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:35.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:35.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:35.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:35.845+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:35.722+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:35.722+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:35.722+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:36.115 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:35.722+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:36.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:36.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:36.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:36.114+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.843 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:36.670+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 581 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:36.843 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:36.678+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:36.843 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:36.678+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:36.843 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:36.678+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:36.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:36.678+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:36.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:36.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:36.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:36.842+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:37.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:37.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:37.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:37.133+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:38.086 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:37.851+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:37.851+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:37.851+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:37.851+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:37.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:37.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:37.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:37.725+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:38.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:38.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:38.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:38.085+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:39.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:38.810+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:39.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:38.810+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:39.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:38.810+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:39.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:38.810+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:39.041 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:38.722+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:39.041 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:38.722+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:39.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:38.722+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:39.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:38.722+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:39.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:39.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:39.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:39.041+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:39.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:39.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:39.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:39.819+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:39.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:39.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:40.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:39.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:40.043 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:39.720+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:40.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:40.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:40.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:40.042+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:41.068 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:40.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:41.068 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:40.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:41.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:40.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:41.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:40.772+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:41.069 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:40.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:41.069 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:40.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:41.069 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:40.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:41.069 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:40.748+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:41.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:41.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:41.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:41.067+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:42.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:41.669+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 608 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:41.793+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:41.793+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:41.793+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:41.793+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:41.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:41.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:41.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:42.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:41.739+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:42.081+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:42.081+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:42.081+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:42.081+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:42.746+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:43.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:42.746+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:43.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:42.746+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:43.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:42.746+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.067 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:42.757+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:43.068 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:42.757+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:43.068 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:42.757+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:43.068 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:42.757+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:43.066+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:43.066+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:43.066+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:43.066+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:43.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:43.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:43.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:43.731+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:43.731+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:43.731+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:43.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:43.731+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:43.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:43.731+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:44.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:44.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:44.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:44.065+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:45.099 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:44.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:45.099 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:44.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:45.100 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:44.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:45.100 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:44.715+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:45.100 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:44.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:45.100 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:44.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:45.100 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:44.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:45.100 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:44.727+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:45.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:45.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:45.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:45.098+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:45.733+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:45.733+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:45.733+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:45.733+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:45.706+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:45.706+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:45.706+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:45.706+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:46.142+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:46.142+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:46.142+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:46.142+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:47.110 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:46.671+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 637 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:47.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:46.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:47.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:46.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:47.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:46.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:47.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:46.688+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:47.111 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:46.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:47.111 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:46.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:47.111 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:46.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:47.111 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:46.707+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:47.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:47.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:47.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:47.109+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:47.667+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:47.668+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:47.668+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:47.668+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:47.712+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:47.712+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:47.712+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:47.712+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:48.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:48.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:48.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:48.136+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:48.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:48.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:48.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:48.754+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:48.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:48.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:48.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:48.630+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:49.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:49.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:49.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:49.125+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:49.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:49.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:49.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:49.781+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:49.676+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:49.676+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:49.676+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:49.676+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:50.127+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:50.127+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:50.127+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:50.127+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:50.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:50.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:50.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:50.755+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:50.690+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:50.690+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:50.690+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:50.690+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:51.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:51.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:51.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:51.164+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:51.671+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 666 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:51.937 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:51.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:51.937 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:51.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:51.938 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:51.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:51.938 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:51.781+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.938 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:51.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:51.938 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:51.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:51.938 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:51.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:51.938 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:51.679+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:52.213+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:52.213+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:52.213+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:52.213+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:52.761+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:52.761+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:52.761+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:52.761+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:52.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:52.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:52.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:52.718+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.529 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:53.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:53.529 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:53.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:53.529 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:53.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:53.529 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:53.199+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.796 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:53.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:53.796 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:53.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:53.796 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:53.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:53.796 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:53.740+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:53.795+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:53.795+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:53.795+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:53.795+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:54.169+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:54.169+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:54.169+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:54.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:54.169+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:55.009 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:54.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:55.009 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:54.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:55.009 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:54.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:55.010 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:54.749+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:55.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:54.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:55.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:54.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:55.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:54.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:55.011 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:54.692+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:55.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:55.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:55.186+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:55.186+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:55.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:55.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:55.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:55.695+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:56.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:55.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:56.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:55.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:56.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:55.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:56.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:55.759+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:56.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:56.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:56.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:56.143+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:57.104 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:46:56.672+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 693 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:46:57.104 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:56.801+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:56.801+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:56.801+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:56.801+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:56.659+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:56.659+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:57.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:56.659+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:57.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:56.659+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:57.103+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:57.103+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:57.103+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:57.103+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:57.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:58.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:57.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:58.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:57.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:58.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:57.695+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:57.784+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:58.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:57.784+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:58.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:57.784+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:58.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:57.784+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:58.055+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:58.055+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:58.055+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:58.055+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:59.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:58.725+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:59.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:58.725+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:59.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:58.726+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:46:59.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:58.726+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:59.056 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:58.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:59.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:58.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:59.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:58.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:46:59.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:58.759+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:59.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:59.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:59.055+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:46:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:46:59.055+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:59.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:00.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:59.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:00.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:59.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:00.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:46:59.766+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.085 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:59.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:00.085 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:59.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:00.085 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:59.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:00.085 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:46:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:46:59.777+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:00.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:00.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:00.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:00.080+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:01.082 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:00.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:01.082 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:00.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:01.082 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:00.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:01.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:00.767+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:01.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:00.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:01.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:00.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:01.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:00.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:01.083 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:00.730+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:01.081+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:01.081+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:01.081+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:01.082+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:02.113 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:01.673+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 735 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:02.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:01.704+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:02.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:01.704+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:02.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:01.704+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:02.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:01.705+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:02.114 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:01.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:02.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:01.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:02.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:01.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:02.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:01.749+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:02.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:02.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:02.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:02.113+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.096 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:02.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:03.096 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:02.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:03.096 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:02.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:03.096 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:02.750+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:02.748+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:03.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:02.748+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:03.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:02.748+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:03.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:02.748+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:03.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:03.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:03.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:03.095+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.790 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:03.761+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:03.790 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:03.761+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:03.790 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:03.761+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:03.791 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:03.761+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:03.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:03.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:03.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:03.790+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:04.141+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:04.141+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:04.141+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:04.141+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:04.755+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:04.755+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:04.755+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:04.755+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:04.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:04.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:04.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:04.823+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:05.141+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:05.141+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:05.141+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:05.141+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:06.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:05.780+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:06.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:05.780+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:06.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:05.780+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:06.114 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:05.780+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:06.114 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:05.866+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:06.114 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:05.866+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:06.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:05.866+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:06.115 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:05.866+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:06.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:06.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:06.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:06.113+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:07.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:06.674+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 763 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:07.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:06.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:07.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:06.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:07.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:06.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:07.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:06.819+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:06.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:06.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:06.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:06.899+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:07.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:07.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:07.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:07.139+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:07.803+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:07.803+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:07.803+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:07.803+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:07.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:07.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:07.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:07.902+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:08.174+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:08.174+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:08.174+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:08.174+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:08.764+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:08.764+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:08.764+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:08.764+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:08.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:08.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:08.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:08.871+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:09.185+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:09.185+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:09.185+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:09.185+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:09.751+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:09.751+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:09.751+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:09.751+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:09.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:09.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:09.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:09.915+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:10.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:10.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:10.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:10.206+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:10.704+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:10.704+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:10.704+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:10.704+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:10.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:10.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:10.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:10.903+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:11.247+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:11.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:11.247+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:11.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:11.247+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:11.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:11.247+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.906 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:11.675+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 791 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:11.907 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:11.671+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:11.907 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:11.671+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:11.907 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:11.671+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:11.907 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:11.671+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:12.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:11.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:12.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:11.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:12.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:11.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:12.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:11.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:12.259+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:12.259+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:12.259+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:12.259+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:12.954 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:12.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:12.954 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:12.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:12.954 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:12.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:12.954 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:12.670+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:12.954+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:13.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:12.954+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:13.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:12.954+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:13.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:12.954+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.530 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:13.256+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:13.530 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:13.256+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:13.530 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:13.256+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:13.530 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:13.256+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:13.694+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:13.694+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:13.694+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:13.694+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:14.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:13.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:14.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:13.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:14.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:13.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:14.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:13.906+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:14.227+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:14.227+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:14.227+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:14.228+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:14.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:14.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:14.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:14.725+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:15.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:14.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:15.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:14.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:15.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:14.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:15.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:14.879+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:15.266+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:15.266+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:15.266+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:15.266+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:15.693+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:15.693+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:15.693+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:15.693+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:15.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:15.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:15.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:15.858+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:16.242+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:16.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:16.242+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:16.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:16.242+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:16.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:16.242+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:16.675+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 817 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:16.801 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:16.711+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:16.802 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:16.711+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:16.802 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:16.711+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:16.802 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:16.711+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:16.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:16.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:16.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:16.840+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:17.274+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:17.274+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:17.274+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:17.274+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:17.677+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:17.677+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:17.677+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:17.677+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:17.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:17.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:17.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:17.799+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:18.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:18.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:18.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:18.246+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:18.683+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:18.683+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:18.683+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:18.683+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:18.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:18.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:18.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:18.823+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:19.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:19.296+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:19.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:19.296+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:19.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:19.296+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:19.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:19.296+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:19.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:19.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:19.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:19.695+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:19.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:19.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:19.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:19.858+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:20.279+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:20.279+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:20.279+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:20.279+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:20.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:20.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:20.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:20.715+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:20.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:20.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:20.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:20.877+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:21.283+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:21.283+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:21.283+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:21.283+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:21.675+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 845 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:22.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:21.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:22.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:21.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:21.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:21.829+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:22.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:21.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:22.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:21.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:22.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:21.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:22.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:21.709+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:22.288+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:22.288+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:22.288+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:22.288+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:22.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:22.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:22.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:22.714+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:22.838+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:22.838+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:22.838+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:22.839+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:23.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:23.338+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:23.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:23.338+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:23.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:23.338+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:23.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:23.338+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:23.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:23.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:23.748+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:23.748+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:23.830+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:23.830+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:23.830+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:23.830+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:24.357+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:24.357+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:24.357+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:24.357+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:24.792+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:24.792+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:24.792+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:24.792+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:24.873+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:24.873+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:24.873+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:24.873+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:25.728 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:25.381+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:25.729 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:25.381+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:25.729 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:25.381+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:25.729 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:25.381+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:25.757+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:25.757+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:25.757+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:25.757+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:25.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:25.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:25.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:25.918+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.678 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:26.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:26.678 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:26.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:26.678 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:26.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:26.678 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:26.384+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:26.676+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 873 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:26.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:26.772+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:26.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:26.772+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:26.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:26.772+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:26.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:26.772+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:26.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:26.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:26.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:26.967+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:27.731 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:27.395+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:27.732 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:27.395+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:27.732 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:27.395+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:27.732 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:27.396+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:27.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:27.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:27.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:27.731+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:27.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:27.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:27.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:27.924+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:28.701 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:28.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:28.701 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:28.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:28.701 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:28.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:28.701 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:28.370+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:28.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:28.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:28.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:28.700+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:28.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:28.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:28.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:28.942+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:29.365+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:29.365+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:29.365+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:29.365+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:29.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:29.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:29.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:29.749+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:29.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:29.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:29.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:29.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:30.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:30.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:30.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:30.337+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:30.780+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:30.780+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:30.780+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:30.780+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:30.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:30.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:30.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:30.872+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:31.639 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:31.374+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:31.639 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:31.374+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:31.639 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:31.374+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:31.639 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:31.374+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:31.677+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 901 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:31.818+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:31.818+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:31.818+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:31.818+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:31.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:31.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:31.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:31.830+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:32.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:32.415+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:32.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:32.415+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:32.797 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:32.415+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:32.797 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:32.416+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:32.840+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:32.840+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:32.840+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:32.840+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:32.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:32.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:32.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:32.796+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:33.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:33.461+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:33.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:33.461+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:33.793 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:33.461+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:33.793 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:33.461+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:33.791+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:33.791+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:33.791+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:33.792+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:33.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:33.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:33.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:33.824+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:34.827 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:34.486+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:34.827 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:34.486+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:34.827 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:34.486+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:34.828 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:34.486+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:34.826+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:34.826+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:34.826+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:34.827+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:34.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:34.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:34.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:34.846+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:35.787 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:35.511+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:35.787 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:35.511+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:35.787 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:35.511+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:35.787 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:35.511+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:35.786+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:35.786+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:35.786+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:35.786+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:35.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:35.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:35.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:35.798+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:36.783 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:36.678+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 927 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:36.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:36.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:36.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:36.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:36.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:36.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:36.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:36.487+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:36.811+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:36.811+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:36.811+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:36.811+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:36.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:36.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:36.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:36.783+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:37.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:37.497+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:37.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:37.497+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:37.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:37.497+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:37.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:37.497+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:37.840+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:37.840+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:37.840+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:37.840+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:37.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:37.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:37.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:37.797+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:38.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:38.451+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:38.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:38.451+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:38.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:38.451+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:38.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:38.451+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:38.887+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:38.887+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:38.887+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:38.887+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:38.788+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:38.788+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:38.788+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:38.789+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:39.809 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:39.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:39.810 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:39.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:39.810 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:39.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:39.810 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:39.441+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:39.841+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:39.841+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:39.841+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:39.841+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:39.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:39.809+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:39.809+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:39.809+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:40.825 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:40.485+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:40.825 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:40.485+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:40.825 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:40.485+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:40.826 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:40.485+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:40.824+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:40.824+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:40.824+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:40.825+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:40.833+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:40.833+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:40.833+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:40.833+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:41.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:41.679+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 955 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:41.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:41.454+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:41.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:41.454+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:41.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:41.454+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:41.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:41.454+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:41.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:41.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:41.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:41.795+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:42.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:41.846+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:42.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:41.846+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:42.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:41.846+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:42.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:41.846+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:42.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:42.473+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:42.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:42.473+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:42.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:42.473+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:42.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:42.473+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:42.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:42.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:42.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:42.813+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:42.864+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:42.864+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:42.864+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:43.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:42.864+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:43.788 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:43.505+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:43.788 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:43.505+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:43.789 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:43.505+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:43.789 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:43.505+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:43.904+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:43.904+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:43.904+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:43.904+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:43.788+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:43.788+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:43.788+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:43.788+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:44.768 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:44.488+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:44.768 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:44.488+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:44.768 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:44.488+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:44.768 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:44.488+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:44.912+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:44.912+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:44.912+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:44.913+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:44.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:44.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:44.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:44.767+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:45.781 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:45.446+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:45.781 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:45.446+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:45.781 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:45.446+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:45.781 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:45.446+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:45.946+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:45.946+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:45.946+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:45.946+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:45.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:45.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:45.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:45.780+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.704 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:46.680+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 983 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:46.704 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:46.442+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:46.704 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:46.442+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:46.704 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:46.442+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:46.704 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:46.442+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:46.798+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:46.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:46.798+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:46.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:46.798+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:46.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:46.798+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:46.977+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:46.977+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:46.977+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:46.977+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:47.751 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:47.422+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:47.752 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:47.422+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:47.752 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:47.422+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:47.752 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:47.422+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:47.940+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:47.940+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:47.940+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:47.940+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:47.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:47.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:47.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:47.751+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:48.713+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:48.713+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:48.713+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:48.713+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:48.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:48.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:48.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:48.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:48.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:48.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:48.468+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:48.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:48.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:48.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:48.960+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:49.735 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:49.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:49.736 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:49.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:49.736 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:49.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:49.736 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:49.445+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:49.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:49.735+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:49.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:49.735+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:49.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:49.735+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:49.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:49.735+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:49.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:49.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:49.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:49.998+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:50.757 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:50.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:50.758 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:50.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:50.758 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:50.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:50.758 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:50.487+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:50.974+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:50.974+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:50.974+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:50.974+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:50.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:50.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:50.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:50.757+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.807 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:51.681+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1011 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:51.807 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:51.491+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:51.491+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:51.491+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:51.491+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:51.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:51.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:51.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:51.808 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:51.709+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:51.997+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:51.997+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:51.997+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:51.997+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:52.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:52.456+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:52.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:52.456+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:52.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:52.456+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:52.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:52.456+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:52.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:52.704+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:52.704+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:52.704+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:52.704+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:53.018+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:53.018+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:53.018+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:53.018+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:53.740 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:53.474+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:53.740 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:53.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:53.740 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:53.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:53.740 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:53.475+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.039 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:53.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:54.039 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:53.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:54.039 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:53.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:54.039 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:53.739+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:47:54.195+0000 7f21d0666700 0 monclient(hunting): authenticate timed out after 300 2024-02-10T06:47:54.197 INFO:teuthology.orchestra.run.smithi073.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T06:47:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:54.039+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:54.039+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:54.039+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:54.039+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:54.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:54.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:54.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:54.730+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:54.496+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:54.496+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:54.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:54.496+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:54.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:54.496+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.981 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:47:55.249 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/mon.smithi073/config 2024-02-10T06:47:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:55.055+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:55.055+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:55.055+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:55.055+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:55.687 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:55.477+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:55.687 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:55.477+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:55.687 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:55.477+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:55.687 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:55.477+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:56.022 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:55.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:56.022 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:55.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:56.022 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:55.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:56.022 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:55.716+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:56.349 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:56.021+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:56.349 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:56.021+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:56.350 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:56.021+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:56.350 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:56.021+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:56.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:56.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:56.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:56.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:56.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:56.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:56.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:56.445+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:57.051 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:47:56.681+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1037 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:47:57.051 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:56.713+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:57.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:56.713+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:57.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:56.713+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:57.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:56.713+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:57.050+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:57.050+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:57.050+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:57.050+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:57.698 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:57.436+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:57.699 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:57.436+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:57.699 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:57.436+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:57.699 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:57.436+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.064 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:57.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:58.064 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:57.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:58.064 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:57.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:58.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:57.698+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:58.064+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:58.064+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:58.064+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:58.064+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:58.429+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:58.429+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:58.429+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:58.429+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:58.675+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:58.675+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:58.675+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:58.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:58.675+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:59.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:59.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:59.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:47:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:47:59.042+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:59.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:59.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:59.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:59.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:59.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:59.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:47:59.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:47:59.416+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:59.636+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:59.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:59.636+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:59.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:59.636+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:47:59.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:47:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:47:59.636+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:00.051+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:00.051+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:00.051+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:00.051+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:00.439+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:00.439+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:00.439+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:00.440+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:00.590+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:00.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:00.590+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:00.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:00.590+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:00.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:00.590+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:01.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:01.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:01.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:01.015+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:01.683 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:01.611+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:01.683 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:01.611+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:01.683 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:01.611+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:01.683 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:01.611+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:01.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:01.391+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:01.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:01.391+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:01.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:01.391+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:01.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:01.391+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.034 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:01.682+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1065 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:02.033+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:02.033+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:02.033+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:02.033+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:02.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:02.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:02.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:02.370+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:02.657+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:03.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:02.657+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:03.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:02.657+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:03.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:02.657+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.344 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:03.040+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:03.345 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:03.040+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:03.345 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:03.040+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:03.345 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:03.040+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:03.344+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:03.344+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:03.344+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:03.344+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.933 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:03.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:03.933 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:03.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:03.934 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:03.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:03.934 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:03.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:04.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:04.086+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:04.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:04.086+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:04.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:04.086+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:04.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:04.087+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:04.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:04.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:04.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:04.364+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:04.635+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:05.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:04.635+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:05.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:04.635+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:05.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:04.635+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:05.100+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:05.100+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:05.100+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:05.101+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:05.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:05.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:05.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:05.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:05.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:05.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:05.410+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.665 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:05.649+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:05.666 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:05.649+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:05.666 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:05.649+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:05.666 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:05.649+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:06.056+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:06.056+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:06.056+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:06.056+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:06.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:06.375+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:06.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:06.375+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:06.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:06.375+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:06.665 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:06.375+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.023 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:06.682+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1093 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:07.023 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:06.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:07.024 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:06.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:07.024 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:06.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:07.024 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:06.663+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.341 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:07.022+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:07.341 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:07.022+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:07.341 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:07.022+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:07.341 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:07.022+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:07.340+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:07.340+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:07.340+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:07.340+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:08.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:07.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:08.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:07.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:08.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:07.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:08.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:07.647+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:08.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:08.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:08.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:08.015+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:08.737 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:08.375+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:08.737 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:08.375+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:08.737 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:08.375+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:08.738 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:08.375+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:08.738 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:08.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:08.738 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:08.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:08.738 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:08.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:08.738 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:08.619+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:09.365 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:09.028+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:09.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:09.028+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:09.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:09.028+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:09.366 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:09.029+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:09.576+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:09.576+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:09.576+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:09.577+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:09.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:09.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:09.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:09.365+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:09.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:09.365+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:09.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:09.365+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.362 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:10.051+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:10.362 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:10.051+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:10.363 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:10.051+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:10.363 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:10.051+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:10.361+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:10.361+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:10.361+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:10.362+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:10.534+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:10.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:10.534+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:10.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:10.534+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:10.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:10.535+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:11.016+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:11.016+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:11.016+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:11.016+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:11.641 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:11.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:11.668 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:11.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:11.668 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:11.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:11.668 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:11.569+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:11.668 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:11.377+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:11.668 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:11.377+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:11.668 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:11.377+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:11.669 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:11.377+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.066 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:11.683+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1137 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:12.065+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:12.065+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:12.065+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:12.065+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:12.415+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:12.415+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:12.415+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:12.415+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:12.589+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:12.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:12.589+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:12.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:12.589+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:12.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:12.589+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:13.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:13.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:13.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:13.079+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:13.423+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:13.423+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:13.423+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:13.424+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:13.568+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:13.568+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:13.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:13.568+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:13.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:13.568+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:14.361 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:14.107+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:14.361 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:14.107+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:14.361 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:14.107+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:14.362 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:14.107+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:14.385+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:14.386+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:14.386+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:14.386+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:14.559+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:14.559+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:14.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:14.559+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:14.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:14.559+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:15.072+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:15.072+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:15.073+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:15.073+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:15.393+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:15.393+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:15.393+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:15.394+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:15.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:15.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:15.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:15.594+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:16.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:16.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:16.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:16.063+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:16.685 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:16.584+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:16.685 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:16.584+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:16.685 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:16.584+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:16.685 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:16.584+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:16.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:16.403+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:16.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:16.403+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:16.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:16.403+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:16.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:16.404+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:17.113 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:16.684+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1180 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:17.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:17.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:17.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:17.113+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:17.631 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:17.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:17.631 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:17.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:17.631 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:17.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:17.631 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:17.416+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:17.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:17.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:17.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:17.630+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:18.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:18.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:18.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:18.131+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:18.369+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:18.369+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:18.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:18.369+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:18.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:18.369+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:19.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:18.678+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:19.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:18.678+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:19.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:18.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:19.114 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:18.679+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:19.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:19.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:19.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:19.113+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:19.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:19.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:19.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:19.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:19.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:19.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:19.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:19.409+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:20.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:19.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:20.118 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:19.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:20.118 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:19.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:20.118 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:19.681+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:20.116+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:20.117+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:20.117+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:20.117+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:20.672 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:20.381+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:20.672 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:20.381+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:20.672 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:20.381+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:20.672 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:20.381+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:20.671+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:21.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:20.671+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:21.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:20.671+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:21.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:20.671+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:21.101+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:21.101+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:21.101+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:21.101+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:21.427+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:21.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:21.427+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:21.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:21.427+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:21.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:21.427+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:22.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:21.685+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1260 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:22.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:21.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:22.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:21.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:22.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:21.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:22.102 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:21.685+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:22.100+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:22.100+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:22.100+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:22.100+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:22.723 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:22.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:22.723 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:22.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:22.724 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:22.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:22.724 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:22.468+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:23.111 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:22.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:23.112 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:22.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:23.112 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:22.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:23.112 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:22.722+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:23.111+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:23.111+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:23.111+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:23.111+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:23.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:23.485+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:23.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:23.485+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:23.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:23.485+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:23.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:23.485+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:24.096 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:23.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:24.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:23.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:24.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:23.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:24.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:23.754+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:24.125+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:24.125+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:24.125+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:24.125+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:24.741 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:24.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:24.741 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:24.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:24.741 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:24.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:24.741 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:24.442+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:24.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:24.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:24.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:24.740+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.429 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:25.164+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:25.429 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:25.164+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:25.429 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:25.164+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:25.429 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:25.164+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.727 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:25.428+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:25.728 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:25.428+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:25.728 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:25.428+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:25.728 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:25.428+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:25.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:25.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:25.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:25.727+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:26.380+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:26.380+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:26.380+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:26.380+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:26.208+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:26.208+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:26.208+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:26.208+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:26.686+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1355 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:26.760+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:26.760+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:26.760+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:26.760+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:27.237+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:27.237+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:27.237+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:27.237+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:27.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:27.380+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:27.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:27.380+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:27.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:27.380+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:27.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:27.380+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:27.770+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:27.770+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:27.770+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:27.771+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:28.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:28.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:28.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:28.230+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:28.352+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:28.352+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:28.352+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:28.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:28.352+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:28.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:28.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:28.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:28.729+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:29.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:29.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:29.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:29.376+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:29.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:29.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:29.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:29.273+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:29.774+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:29.774+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:29.774+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:29.774+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:30.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:30.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:30.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:30.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:30.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:30.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:30.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:30.233+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:30.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:30.413+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:30.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:30.413+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:30.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:30.413+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:30.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:30.414+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:30.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:30.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:30.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:30.729+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:31.398+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:31.398+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:31.398+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:31.399+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:31.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:31.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:31.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:31.265+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:31.687+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1450 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:31.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:31.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:31.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:31.701+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:32.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:32.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:32.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:32.217+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:32.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:32.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:32.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:32.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:32.376+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:32.750+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:32.750+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:32.750+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:32.750+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:33.250+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:33.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:33.250+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:33.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:33.250+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:33.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:33.251+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.536 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:33.350+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:33.536 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:33.350+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:33.536 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:33.350+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:33.536 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:33.350+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:33.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:33.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:33.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:33.714+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:34.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:34.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:34.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:34.229+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:34.320+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:34.320+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:34.320+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:34.321+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:34.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:34.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:34.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:34.749+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:35.248+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:35.248+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:35.248+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:35.249+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:35.321+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:35.321+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:35.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:35.321+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:35.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:35.322+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:35.716+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:35.716+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:35.716+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:35.716+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:36.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:36.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:36.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:36.265+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:36.330+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:36.330+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:36.330+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:36.330+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:36.687+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1544 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:36.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:36.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:36.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:36.766+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:37.281+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:37.281+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:37.281+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:37.281+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:37.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:37.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:37.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:37.310+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:37.719+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:37.719+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:37.719+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:37.719+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:38.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:38.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:38.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:38.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:38.319+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:38.319+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:38.319+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:38.319+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:38.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:38.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:38.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:38.696+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:39.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:39.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:39.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:39.300+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:39.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:39.301+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:39.301+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:39.301+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:39.301+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:39.728+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:39.728+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:39.728+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:39.729+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:40.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:40.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:40.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:40.256+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:40.293+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:40.293+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:40.293+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:40.293+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:40.768+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:40.768+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:40.768+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:40.768+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:41.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:41.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:41.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:41.297+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:41.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:41.325+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:41.325+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:41.325+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:41.325+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:41.688+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1640 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:41.758+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:41.758+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:41.758+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:41.758+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:42.312+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:42.312+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:42.312+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:42.312+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:42.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:42.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:42.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:42.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:42.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:42.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:42.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:42.370+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:42.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:42.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:42.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:42.795+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:43.536 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:43.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:43.536 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:43.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:43.536 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:43.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:43.536 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:43.275+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:43.536 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:43.388+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:43.537 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:43.388+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:43.537 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:43.388+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:43.537 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:43.388+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:43.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:43.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:43.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:43.767+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:44.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:44.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:44.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:44.291+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:44.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:44.432+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:44.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:44.432+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:44.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:44.432+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:44.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:44.432+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:44.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:44.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:44.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:44.795+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.603 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:45.401+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:45.604 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:45.401+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:45.604 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:45.401+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:45.604 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:45.401+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.604 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:45.289+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:45.604 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:45.289+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:45.604 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:45.289+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:45.605 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:45.289+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:45.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:45.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:45.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:45.790+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:46.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:46.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:46.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:46.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:46.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:46.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:46.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:46.316+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:46.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:46.401+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:46.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:46.401+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:46.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:46.401+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:46.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:46.401+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:46.688+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1736 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:46.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:46.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:46.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:46.766+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:47.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:47.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:47.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:47.358+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:47.421+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:47.421+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:47.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:47.421+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:47.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:47.421+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:47.801+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:47.801+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:47.801+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:47.801+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:48.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:48.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:48.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:48.310+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:48.443+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:48.443+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:48.443+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:48.443+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:48.806+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:48.806+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:48.806+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:48.806+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:49.313+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:49.313+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:49.313+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:49.313+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:49.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:49.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:49.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:49.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:49.409+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:49.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:49.409+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:49.806+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:49.806+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:49.806+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:49.806+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:50.326+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:50.326+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:50.326+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:50.327+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:50.440+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:50.440+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:50.440+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:50.441+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:50.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:50.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:50.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:50.786+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:51.313+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:51.313+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:51.313+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:51.313+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:51.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:51.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:51.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:51.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:51.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:51.489+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:52.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:51.690+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1832 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:51.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:51.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:51.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:51.743+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:52.296+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:52.296+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:52.296+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:52.296+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:52.477+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:52.477+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:52.477+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:52.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:52.477+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:52.765+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:52.765+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:52.765+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:52.765+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:53.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:53.516+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:53.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:53.516+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:53.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:53.516+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:53.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:53.516+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:53.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:53.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:53.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:53.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:53.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:53.300+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:53.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:53.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:53.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:53.757+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:54.513+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:54.513+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:54.513+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:54.513+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:54.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:54.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:54.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:54.277+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:54.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:54.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:54.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:55.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:54.771+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:55.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:55.481+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:55.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:55.481+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:55.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:55.481+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:55.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:55.481+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:55.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:55.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:55.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:55.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:55.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:55.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:55.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:55.308+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:55.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:55.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:55.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:55.791+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:56.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:56.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:56.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:56.285+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:56.491+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:56.491+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:56.491+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:56.491+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:57.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:48:56.691+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 1926 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:48:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:56.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:56.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:56.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:56.777+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:57.332+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:57.332+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:57.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:57.332+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:57.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:57.332+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:57.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:57.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:57.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:57.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:57.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:57.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:57.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:57.475+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:57.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:57.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:57.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:57.802+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:58.495 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:58.290+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:58.495 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:58.290+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:58.496 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:58.290+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:58.496 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:58.290+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:58.842 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:58.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:58.842 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:58.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:58.843 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:58.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:58.843 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:58.495+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:58.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:58.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:58.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:48:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:58.842+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:59.520 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:59.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:59.521 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:59.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:59.521 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:59.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:48:59.521 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:48:59.269+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:59.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:59.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:59.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:59.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:59.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:59.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:48:59.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:48:59.519+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:59.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:59.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:59.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:48:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:48:59.839+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:00.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:00.529+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:00.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:00.529+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:00.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:00.529+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:00.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:00.530+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:00.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:00.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:00.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:00.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:00.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:00.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:00.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:00.297+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:00.864+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:00.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:00.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:00.865+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:01.488+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:01.488+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:01.488+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:01.488+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:01.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:01.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:01.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:01.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:01.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:01.297+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:01.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:01.692+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2021 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:02.344 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:01.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:02.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:01.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:02.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:01.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:02.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:01.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:02.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:02.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:02.344+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:02.344+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:02.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:02.447+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:02.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:02.447+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:02.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:02.447+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:02.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:02.447+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:03.298 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:02.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:03.298 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:02.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:03.298 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:02.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:03.298 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:02.931+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:03.452+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:03.452+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:03.452+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:03.452+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:03.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:03.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:03.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:03.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:03.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:03.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:03.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:03.297+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:04.295 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:03.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:04.295 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:03.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:04.295 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:03.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:04.295 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:03.932+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:04.423+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:04.423+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:04.423+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:04.423+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:04.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:04.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:04.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:04.295+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:05.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:04.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:05.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:04.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:05.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:04.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:05.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:04.981+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:05.414+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:05.414+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:05.414+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:05.414+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:05.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:05.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:05.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:05.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:05.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:05.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:05.276+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:06.259 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:05.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:06.259 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:05.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:06.259 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:05.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:06.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:05.957+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:06.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:06.455+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:06.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:06.455+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:06.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:06.455+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:06.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:06.455+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:06.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:06.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:06.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:06.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:06.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:06.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:06.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:07.105 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:06.692+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2117 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:07.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:06.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:07.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:06.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:07.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:06.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:07.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:06.909+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:07.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:07.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:07.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:07.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:07.431+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:07.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:07.431+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:07.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:07.431+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:07.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:07.431+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:08.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:07.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:08.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:07.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:08.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:07.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:08.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:07.930+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:08.240+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:08.240+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:08.240+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:08.240+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:08.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:08.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:08.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:08.475+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:08.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:08.475+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:09.218 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:08.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:09.218 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:08.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:09.219 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:08.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:09.219 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:08.930+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:09.497 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:09.218+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:09.498 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:09.218+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:09.498 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:09.218+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:09.498 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:09.218+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:09.497+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:09.497+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:09.497+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:09.497+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:10.209 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:09.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:10.209 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:09.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:10.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:09.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:10.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:09.936+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:10.508 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:10.209+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:10.508 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:10.209+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:10.508 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:10.209+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:10.508 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:10.209+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:10.508+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:10.508+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:10.508+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:10.508+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:11.242 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:10.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:11.242 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:10.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:11.242 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:10.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:11.242 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:10.961+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:11.531 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:11.241+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:11.531 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:11.241+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:11.531 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:11.241+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:11.532 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:11.241+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:11.821 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:11.693+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2213 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:11.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:11.530+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:11.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:11.530+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:11.822 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:11.530+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:11.822 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:11.530+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:11.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:11.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:11.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:11.962+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:12.541 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:12.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:12.541 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:12.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:12.542 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:12.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:12.542 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:12.283+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:12.540+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:12.540+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:12.540+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:12.540+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:13.244 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:12.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:13.244 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:12.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:13.244 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:12.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:13.244 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:12.959+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:13.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:13.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:13.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:13.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:13.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:13.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:13.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:13.244+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:13.575+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:13.575+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:13.575+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:13.575+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:14.276 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:13.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:14.276 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:13.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:14.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:13.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:14.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:13.969+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:14.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:14.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:14.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:14.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:14.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:14.276+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:14.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:14.276+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:14.535+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:14.535+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:14.535+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:14.535+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:15.259 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:14.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:15.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:14.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:15.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:14.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:15.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:14.975+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:15.501+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:15.501+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:15.501+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:15.501+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:15.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:15.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:15.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:15.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:16.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:16.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:16.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:16.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:16.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:16.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:16.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:16.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:16.322 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:16.237+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:16.322 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:16.237+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:16.322 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:16.237+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:16.322 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:16.237+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:16.512+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:16.512+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:16.512+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:16.512+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:17.020 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:16.694+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2306 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:17.262+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:17.262+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:17.262+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:17.262+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.020+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:17.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:17.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:17.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:17.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:17.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:17.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:17.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:17.494+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:18.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:18.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:18.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:18.234+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:17.989+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:18.470+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:18.470+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:18.470+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:18.470+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:19.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:18.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:19.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:18.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:19.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:18.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:19.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:18.983+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:19.252+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:19.252+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:19.252+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:19.252+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:19.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:19.455+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:19.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:19.455+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:19.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:19.455+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:19.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:19.455+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:20.231+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:20.231+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:20.231+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:20.232+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:20.013+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:20.013+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:20.013+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:20.013+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:20.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:20.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:20.416+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:20.417+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:21.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:21.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:21.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:21.220+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:21.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:21.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:21.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:21.022+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:21.707 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:21.695+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2402 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:21.708 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:21.453+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:21.708 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:21.453+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:21.708 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:21.453+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:21.708 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:21.453+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:22.242+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:22.242+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:22.242+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:22.242+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.005+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.005+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.005+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.005+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:22.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:22.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:22.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:22.495+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:23.280 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:23.280 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:23.280 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:23.281 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:22.979+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:23.482+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:23.482+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:23.482+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:23.482+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:23.279+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:23.279+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:23.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:23.279+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:23.539 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:23.279+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:24.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:23.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:24.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:23.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:24.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:23.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:24.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:23.957+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:24.503+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:24.503+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:24.503+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:24.503+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:24.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:24.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:24.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:24.322+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:25.305 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:24.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:25.305 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:24.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:25.305 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:24.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:25.305 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:24.935+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:25.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:25.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:25.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:25.304+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:25.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:25.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:25.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:25.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:25.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:25.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:25.489+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:26.299 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:25.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:26.299 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:25.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:26.299 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:25.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:26.299 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:25.922+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:26.474+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:26.474+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:26.474+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:26.474+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:26.298+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:26.298+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:26.298+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:26.298+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:26.696+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2498 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:26.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:26.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:26.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:26.929+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:27.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:27.509+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:27.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:27.509+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:27.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:27.509+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:27.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:27.509+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:27.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:27.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:27.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:27.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:27.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:27.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:27.261+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:28.311 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:27.895+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:28.311 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:27.895+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:28.311 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:27.895+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:28.311 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:27.895+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:28.559+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:28.559+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:28.559+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:28.559+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:28.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:28.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:28.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:28.311+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:29.344 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:28.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:29.344 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:28.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:29.344 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:28.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:29.344 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:28.883+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:29.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:29.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:29.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:29.343+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:29.562+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:29.562+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:29.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:29.562+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:29.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:29.563+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:29.843+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:29.843+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:29.843+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:29.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:30.544+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:30.544+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:30.544+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:30.545+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:30.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:30.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:30.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:30.305+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:30.863+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:30.863+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:30.863+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:30.863+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:31.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:31.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:31.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:31.301+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:31.514+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:31.514+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:31.514+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:31.515+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:31.698+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2594 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:31.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:31.841+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:31.841+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:31.841+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:32.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:32.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:32.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:32.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:32.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:32.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:32.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:32.346+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:32.858 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:32.515+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:32.859 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:32.515+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:32.859 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:32.515+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:32.859 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:32.515+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:32.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:32.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:32.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:32.858+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:33.499 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:33.328+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:33.499 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:33.328+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:33.499 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:33.328+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:33.499 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:33.328+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:33.498+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:33.498+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:33.498+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:33.498+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:34.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:33.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:34.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:33.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:34.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:33.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:34.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:33.904+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:34.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:34.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:34.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:34.367+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:34.532+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:34.532+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:34.532+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:34.532+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:34.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:34.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:34.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:34.865+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:35.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:35.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:35.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:35.334+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:35.532+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:35.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:35.532+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:35.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:35.532+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:35.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:35.532+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:35.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:35.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:35.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:35.840+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:36.347+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:36.347+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:36.347+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:36.347+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:36.526+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:36.526+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:36.526+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:36.526+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:36.699+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2687 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:36.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:36.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:36.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:36.872+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:37.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:37.366+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:37.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:37.366+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:37.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:37.366+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:37.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:37.367+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:37.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:37.492+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:37.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:37.492+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:37.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:37.492+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:37.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:37.492+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:37.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:37.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:37.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:37.846+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:38.490+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:38.490+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:38.490+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:38.490+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:38.396+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:38.396+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:38.396+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:38.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:38.396+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:38.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:38.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:38.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:38.813+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:39.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:39.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:39.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:39.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:39.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:39.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:39.781 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:39.425+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:39.781 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:39.479+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:39.782 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:39.479+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:39.782 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:39.479+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:39.782 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:39.479+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:39.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:39.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:39.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:39.780+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:40.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:40.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:40.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:40.432+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:40.456+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:40.456+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:40.456+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:40.798 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:40.456+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:40.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:40.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:40.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:40.797+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:41.701 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:41.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:41.701 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:41.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:41.701 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:41.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:41.701 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:41.405+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:41.701 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:41.465+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:41.702 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:41.465+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:41.702 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:41.465+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:41.702 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:41.465+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:41.700+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2782 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:41.832+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:41.832+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:41.832+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:41.832+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:42.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:42.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:42.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:42.438+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:42.460+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:42.460+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:42.460+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:42.868 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:42.460+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:42.867+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:42.867+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:42.867+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:42.867+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:43.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:43.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:43.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:43.404+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:43.429+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:43.429+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:43.429+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:43.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:43.429+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:43.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:43.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:43.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:43.879+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:44.407+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:44.407+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:44.407+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:44.407+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:44.413+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:44.413+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:44.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:44.413+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:44.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:44.413+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:44.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:44.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:44.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:44.881+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:45.448+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:45.448+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:45.448+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:45.449+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:45.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:45.457+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:45.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:45.457+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:45.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:45.457+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:45.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:45.458+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:45.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:45.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:45.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:45.914+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:46.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:46.701+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2877 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:46.713 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:46.459+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:46.459+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:46.459+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:46.460+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:46.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:46.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:46.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:46.714 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:46.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:46.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:46.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:46.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:46.877+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:47.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:47.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:47.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:47.494+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:47.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:47.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:47.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:47.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:47.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:47.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:47.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:47.468+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:47.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:47.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:47.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:47.896+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:48.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:48.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:48.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:48.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:48.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:48.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:48.500+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:48.917 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:48.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:48.917 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:48.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:48.917 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:48.487+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:48.917 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:48.487+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:48.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:48.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:48.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:48.938+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:49.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:49.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:49.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:49.453+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:49.464+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:49.464+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:49.464+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:49.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:49.464+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:49.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:49.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:49.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:49.928+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:50.487+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:50.487+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:50.487+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:50.487+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:50.438+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:50.438+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:50.438+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:50.438+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:50.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:50.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:50.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:50.896+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:51.827 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:51.702+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 2973 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:51.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:51.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:51.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:51.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:51.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:51.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:51.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:51.495+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:51.828 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:51.478+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:51.828 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:51.478+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:51.828 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:51.478+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:51.828 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:51.479+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:51.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:51.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:51.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:52.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:51.871+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:52.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:52.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:52.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:52.453+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:52.492+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:52.492+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:52.492+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:52.492+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:53.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:52.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:53.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:52.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:53.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:52.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:53.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:52.885+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:53.839 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:53.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:53.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:53.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:53.495+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:53.498+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:53.498+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:53.498+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:53.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:53.498+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:53.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:53.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:53.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:53.839+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:54.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:54.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:54.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:54.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:54.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:54.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:54.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:54.494+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:54.801 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:54.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:54.801 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:54.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:54.801 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:54.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:54.801 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:54.515+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:54.800+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:54.800+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:54.800+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:54.800+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:55.847 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:55.539+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:55.847 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:55.539+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:55.847 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:55.539+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:55.848 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:55.539+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:55.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:55.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:55.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:55.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:55.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:55.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:55.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:55.509+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:55.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:55.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:55.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:55.847+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:56.514 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:56.502+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:56.514 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:56.502+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:56.514 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:56.502+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:56.514 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:56.502+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:56.824 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:49:56.703+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3066 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:49:56.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:56.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:56.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:56.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:56.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:56.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:56.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:56.513+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:56.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:56.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:56.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:56.824+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:57.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:57.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:57.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:57.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:57.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:57.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:57.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:57.480+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:57.819 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:57.478+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:57.819 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:57.478+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:57.820 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:57.478+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:57.820 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:57.478+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:57.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:57.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:57.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:57.819+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:58.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:58.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:58.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:58.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:58.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:58.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:58.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:58.454+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:58.844 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:58.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:58.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:58.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:58.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:58.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:58.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:58.430+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:58.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:58.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:58.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:49:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:58.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:59.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:59.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:59.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:49:59.413+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:59.390+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:59.390+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:59.390+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:49:59.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:49:59.390+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:59.886+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:59.886+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:00.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:59.886+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:00.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:49:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:49:59.886+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:00.408+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:00.408+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:00.408+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:00.408+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:00.377+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:00.377+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:00.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:00.377+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:00.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:00.377+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:00.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:00.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:00.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:00.912+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:01.649 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:01.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:01.649 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:01.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:01.649 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:01.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:01.649 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:01.399+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:01.650 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:01.388+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:01.650 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:01.388+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:01.650 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:01.388+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:01.650 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:01.388+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:01.906 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:01.704+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3161 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:01.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:01.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:01.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:01.906+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:02.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:02.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:02.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:02.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:02.375+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:02.387+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:02.387+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:02.387+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:02.388+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:03.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:02.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:03.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:02.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:03.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:02.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:03.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:02.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:03.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:03.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:03.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:03.369+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:03.340+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:03.340+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:03.340+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:03.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:03.340+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:04.300 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:03.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:04.300 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:03.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:04.301 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:03.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:04.301 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:03.893+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:04.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:04.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:04.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:04.392+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:04.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:04.299+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:04.299+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:04.299+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:04.299+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:04.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:04.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:04.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:04.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:05.415+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:05.415+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:05.415+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:05.415+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:05.346+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:05.346+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:05.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:05.346+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:05.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:05.346+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:05.862+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:05.862+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:05.862+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:05.862+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:06.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:06.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:06.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:06.399+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:06.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:06.343+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:06.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:06.343+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:06.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:06.343+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:06.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:06.343+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:07.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:06.704+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3257 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:06.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:06.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:06.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:06.828+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:07.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:07.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:07.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:07.356+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:07.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:07.368+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:07.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:07.368+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:07.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:07.368+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:07.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:07.368+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:07.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:07.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:07.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:07.828+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:08.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:08.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:08.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:08.331+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:08.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:08.379+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:08.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:08.379+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:08.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:08.379+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:08.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:08.379+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:09.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:08.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:09.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:08.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:09.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:08.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:09.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:08.872+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:09.363+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:09.363+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:09.363+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:09.363+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:09.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:09.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:09.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:09.336+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:09.833+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:09.833+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:09.833+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:09.833+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:10.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:10.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:10.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:10.358+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:10.305+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:10.305+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:10.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:10.305+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:10.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:10.305+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:10.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:10.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:10.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:10.850+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:11.617 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:11.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:11.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:11.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:11.336+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:11.285+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:11.286+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:11.286+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:11.618 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:11.286+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:12.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:11.706+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3351 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:11.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:11.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:11.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:11.880+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:12.332+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:12.332+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:12.332+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:12.332+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:12.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:12.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:12.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:12.311+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:13.284 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:12.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:13.285 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:12.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:13.285 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:12.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:13.285 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:12.924+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:13.541 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:13.284+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:13.541 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:13.284+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:13.541 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:13.284+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:13.542 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:13.284+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:13.542 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:13.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:13.542 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:13.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:13.542 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:13.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:13.542 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:13.284+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:14.259 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:13.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:14.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:13.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:14.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:13.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:14.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:13.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:14.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:14.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:14.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:14.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:14.259+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:14.259+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:14.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:14.259+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:14.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:14.259+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:15.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:14.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:15.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:14.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:15.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:14.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:15.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:14.988+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:15.249+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:15.249+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:15.249+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:15.249+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:15.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:15.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:15.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:15.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:15.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:15.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:15.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:16.258 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:15.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:16.258 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:15.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:16.258 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:15.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:16.258 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:15.999+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:16.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:16.242+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:16.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:16.242+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:16.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:16.242+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:16.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:16.243+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:16.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:16.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:16.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:16.258+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:16.997 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:16.707+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3445 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:17.238+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:17.238+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:17.238+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:17.238+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:17.200+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:17.200+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:17.200+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:17.201+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:16.997+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:16.997+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:16.997+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:16.997+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:18.199 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:17.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:18.199 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:17.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:18.199 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:17.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:18.200 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:17.948+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:18.198+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:18.198+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:18.198+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:18.198+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:18.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:18.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:18.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:18.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:18.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:18.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:18.233+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:19.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:19.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:19.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:19.197+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:19.183+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:19.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:19.183+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:19.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:19.183+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:19.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:19.183+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:19.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:18.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:19.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:18.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:19.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:18.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:19.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:18.980+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:20.162+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:20.162+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:20.162+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:20.162+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:20.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:20.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:20.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:20.137+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:19.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:19.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:19.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:19.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:21.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:21.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:21.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:21.152+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:21.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:21.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:21.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:21.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:21.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:21.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:21.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:21.112+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:21.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:20.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:21.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:20.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:21.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:20.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:21.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:20.917+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:21.936 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:21.708+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3539 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:22.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:22.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:22.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:22.177+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:22.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:22.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:22.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:22.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:22.122+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:22.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:21.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:22.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:21.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:22.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:21.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:22.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:21.936+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:23.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:23.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:23.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:23.146+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:23.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:23.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:23.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:23.092+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:22.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:22.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:22.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:22.913+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:24.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:24.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:24.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:24.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:24.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:24.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:24.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:24.087+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:24.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:23.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:24.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:23.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:24.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:23.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:24.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:23.950+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:24.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:24.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:24.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:24.184+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:25.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:25.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:25.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:25.173+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:25.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:25.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:25.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:25.120+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:24.997+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:24.997+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:24.997+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:24.997+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:26.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:26.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:26.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:26.201+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:26.128+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:26.128+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:26.128+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:26.128+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:26.030+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:26.030+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:26.030+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:26.030+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:26.977 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:26.709+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3635 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:27.166+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:27.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:27.166+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:27.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:27.166+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:27.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:27.166+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:27.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:27.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:27.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:27.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:27.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:27.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:27.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:27.031+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:27.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:27.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:27.374 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:27.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:27.374 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:27.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:27.374 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:27.218+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:28.182+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:28.182+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:28.182+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:28.182+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:28.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:28.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:28.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:28.153+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:28.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:28.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:28.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:28.055+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:29.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:29.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:29.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:29.217+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:29.176+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:29.176+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:29.176+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:29.176+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:29.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:29.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:29.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:29.092+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:30.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:30.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:30.265+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:30.265+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:30.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:30.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:30.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:30.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:30.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:30.161+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:30.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:30.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:30.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:30.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:30.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:30.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:30.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:30.124+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:31.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:31.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:31.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:31.247+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:31.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:31.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:31.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:31.135+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:31.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:31.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:31.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:31.112+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:32.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:31.710+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3730 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:32.369 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:32.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:32.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:32.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:32.257+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:32.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:32.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:32.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:32.090+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:32.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:32.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:32.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:32.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:32.101+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:33.142+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:33.142+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:33.142+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:33.142+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:33.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:33.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:33.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:33.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:33.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:33.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:33.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:33.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:33.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:33.060+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:34.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:34.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:34.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:34.191+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:34.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:34.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:34.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:34.046+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:34.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:34.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:34.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:34.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:34.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:34.114+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:35.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:35.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:35.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:35.159+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:35.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:35.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:35.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:35.043+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:35.065+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:35.065+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:35.065+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:35.065+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:36.145+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:36.145+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:36.145+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:36.145+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:36.075+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:36.075+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:36.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:36.075+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:36.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:36.075+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:36.097+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:36.097+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:36.097+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:36.097+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:37.057 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:36.711+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3825 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:37.183+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:37.183+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:37.183+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:37.183+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:37.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:37.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:37.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:37.057+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:37.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:37.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:37.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:37.118+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:38.186+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:38.186+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:38.186+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:38.186+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:38.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:38.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:38.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:38.029+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:38.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:38.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:38.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:38.166+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:39.235+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:39.235+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:39.235+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:39.235+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.016+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.016+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.016+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.016+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:39.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:39.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:39.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:39.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:39.120+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:40.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:40.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:40.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:40.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:40.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:40.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:40.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:40.213+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:39.990+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:40.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:40.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:40.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:40.160+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:41.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:41.187+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:41.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:41.187+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:41.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:41.187+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:41.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:41.187+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.009+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:41.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:41.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:41.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:41.118+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:41.992 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:41.712+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 3920 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:42.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:42.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:42.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:42.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:41.992+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:42.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:42.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:42.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:42.189+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:42.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:42.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:42.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:42.338 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:42.118+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:43.113 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:43.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:43.113 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:43.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:43.113 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:43.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:43.114 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:43.019+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:43.228+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:43.228+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:43.228+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:43.228+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:43.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:43.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:43.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:43.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:43.112+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:44.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:44.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:44.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:44.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:44.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:44.197+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:44.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:44.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:44.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:44.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:44.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:44.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:44.054+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:44.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:44.082+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:44.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:44.082+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:44.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:44.082+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:44.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:44.082+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:45.232+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:45.232+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:45.232+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:45.232+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:45.056+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:45.056+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:45.056+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:45.056+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:45.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:45.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:45.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:45.083+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:46.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:46.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:46.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:46.256+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:46.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:46.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:46.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:46.034+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:46.105+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:46.105+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:46.105+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:46.105+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:47.043 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:46.713+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4017 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:47.307 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:47.105+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:47.307 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:47.105+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:47.307 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:47.105+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:47.307 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:47.105+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:47.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:47.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:47.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:47.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:47.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:47.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:47.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:47.042+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:47.306+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:48.273 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:48.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:48.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:48.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:48.110+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:48.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:48.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:48.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:48.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:48.019+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:48.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:48.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:48.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:48.273+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:49.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:49.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:49.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:49.036+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:49.070+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:49.070+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:49.070+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:49.070+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:49.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:49.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:49.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:49.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:49.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:49.269+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:50.288 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:50.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:50.288 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:50.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:50.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:50.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:50.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:50.104+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:50.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:50.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:50.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:50.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:50.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:50.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:50.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:50.034+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:50.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:50.287+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:50.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:50.287+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:50.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:50.287+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:50.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:50.288+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:51.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:51.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:51.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:51.112+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:51.112+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:51.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:51.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:51.067+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:51.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:51.068+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:51.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:51.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:51.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:51.336+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:52.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:51.715+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4112 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:52.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:52.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:52.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:52.115+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:52.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:52.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:52.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:52.114+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:52.370+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:52.370+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:52.370+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:52.371+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:53.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:53.103+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:53.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:53.103+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:53.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:53.103+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:53.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:53.103+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:53.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:53.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:53.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:53.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:53.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:53.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:53.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:53.096+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:53.355+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:53.355+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:53.355+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:53.355+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:54.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:54.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:54.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:54.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:54.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:54.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:54.341 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:54.051+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:54.342 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:54.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:54.342 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:54.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:54.342 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:54.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:54.342 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:54.124+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:54.340+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:54.340+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:54.340+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:54.340+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:55.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:55.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:55.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:55.139+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:55.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:55.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:55.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:55.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:55.062+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:55.321+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:55.321+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:55.321+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:55.321+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:56.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:56.042+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:56.042+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:56.042+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:56.042+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:56.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:56.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:56.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:56.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:56.091+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:56.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:56.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:56.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:56.334+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:57.033 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:50:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:50:56.716+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4206 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:50:57.323 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:57.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:57.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:57.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:57.111+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:57.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:57.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:57.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:57.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:57.033+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:57.323+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:57.323+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:57.323+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:57.323+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:58.366 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:58.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:58.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:58.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:58.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:58.148+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:58.148+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:58.148+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:58.367 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:58.148+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:58.365+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:58.365+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:58.365+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:58.366+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:59.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:59.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:59.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:50:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:50:59.138+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:50:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.027+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:50:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:59.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:59.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:59.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:50:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:50:59.374+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:00.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:00.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:00.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:00.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:50:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:50:59.989+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:00.336 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:00.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:00.336 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:00.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:00.336 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:00.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:00.336 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:00.155+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:00.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:00.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:00.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:00.335+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:01.001+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:01.001+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:01.001+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:01.001+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:01.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:01.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:01.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:01.199+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:01.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:01.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:01.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:01.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:01.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:02.032 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:01.717+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4305 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:02.286 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:02.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:02.286 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:02.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:02.286 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:02.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:02.286 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:02.031+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:02.287 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:02.193+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:02.287 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:02.193+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:02.287 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:02.193+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:02.287 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:02.193+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:02.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:02.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:02.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:02.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:03.181+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:03.181+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:03.181+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:03.182+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:03.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:03.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:03.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:03.245+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.028+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.028+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.028+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.028+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.982+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.982+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.982+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:03.982+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:04.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:04.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:04.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:04.245+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:04.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:04.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:04.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:04.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:04.137+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:05.293 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:05.129+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:05.293 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:05.129+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:05.293 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:05.129+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:05.293 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:05.129+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:05.293 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:04.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:05.293 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:04.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:05.294 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:04.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:05.294 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:04.960+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:05.292+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:05.292+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:05.292+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:05.292+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:06.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:06.162+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:06.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:06.162+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:06.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:06.162+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:06.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:06.162+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:06.292 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:05.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:06.293 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:05.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:06.293 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:05.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:06.293 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:05.967+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:06.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:06.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:06.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:06.291+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:07.011 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:06.718+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4402 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:07.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:07.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:07.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:07.161+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:07.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:07.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:07.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:07.254+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.011+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:08.151+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:08.151+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:08.151+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:08.151+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:08.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:07.972+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:08.225+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:08.225+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:08.225+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:08.225+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:09.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:09.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:09.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:09.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:09.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:09.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:09.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:09.116+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:09.272 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:08.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:09.272 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:08.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:09.272 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:08.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:09.272 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:08.953+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:09.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:09.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:09.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:09.270+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:10.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:10.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:10.137+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:10.137+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:10.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:10.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:10.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:10.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:10.297 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:10.003+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:10.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:10.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:10.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:10.295+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:11.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:11.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:11.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:11.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:11.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:11.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:11.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:11.124+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:11.310 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:11.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:11.310 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:11.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:11.311 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:11.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:11.311 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:11.021+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:11.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:11.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:11.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:11.309+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:12.067 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:11.719+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4498 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:12.066+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:12.066+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:12.066+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:12.066+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:12.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:12.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:12.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:12.283+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:12.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:12.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:12.133+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:12.133+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:13.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:13.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:13.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:13.111+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:13.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:13.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:13.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:13.245+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:13.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:13.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:13.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:13.107+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:14.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:14.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:14.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:14.081+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:14.219+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:14.219+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:14.219+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:14.219+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:14.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:14.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:14.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:14.058+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:15.090+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:15.090+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:15.090+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:15.090+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:15.190+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:15.190+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:15.190+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:15.190+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:15.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:15.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:15.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:15.035+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:16.099+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:16.099+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:16.099+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:16.099+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:16.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:16.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:16.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:16.193+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:16.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:16.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:16.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:16.083+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:17.113 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:16.721+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4598 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:17.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:17.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:17.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:17.113+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:17.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:17.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:17.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:17.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:17.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:17.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:17.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:17.155+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:17.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:17.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:17.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:17.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:17.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:17.120+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:17.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:17.120+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:18.624 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:18.132+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:18.624 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:18.132+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:18.624 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:18.132+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:18.624 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:18.132+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:18.625 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:18.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:18.625 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:18.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:18.625 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:18.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:18.626 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:18.130+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:18.626 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:18.160+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:18.626 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:18.160+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:18.626 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:18.161+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:18.626 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:18.161+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:19.096+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:19.096+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:19.096+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:19.097+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:19.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:19.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:19.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:19.202+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:19.115+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:19.115+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:19.115+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:19.115+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:20.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:20.130+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:20.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:20.130+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:20.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:20.130+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:20.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:20.131+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:20.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:20.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:20.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:20.130+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:20.241+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:20.241+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:20.241+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:20.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:20.241+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:21.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:21.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:21.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:21.087+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:21.084+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:21.084+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:21.084+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:21.084+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:21.208+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:21.208+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:21.208+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:21.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:21.208+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:22.039 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:21.722+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4694 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:22.217+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:22.217+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:22.217+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:22.218+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:22.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:22.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:22.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:22.057+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:22.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:22.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:22.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:22.038+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:23.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:23.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:23.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:23.057+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:23.184+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:23.184+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:23.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:23.184+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:23.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:23.184+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:23.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:23.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:23.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:23.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:23.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:23.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:23.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:23.008+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:24.105+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:24.105+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:24.105+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:24.105+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:24.221+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:24.221+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:24.221+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:24.221+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:24.042+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:24.042+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:24.042+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:24.042+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:25.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:25.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:25.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:25.345 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.008+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:25.345 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:25.058+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:25.058+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:25.058+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:25.058+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:25.188+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:25.188+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:25.188+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:25.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:25.188+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:26.009+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:26.009+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:26.009+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:26.009+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:25.983+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:26.140+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:26.140+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:26.140+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:26.140+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:27.043 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:26.722+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4804 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:27.043 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:26.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:27.043 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:26.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:27.043 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:26.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:27.043 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:26.960+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:27.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:27.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:27.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:27.042+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:27.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:27.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:27.138+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:27.138+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:28.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:28.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:28.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:28.146+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:28.087+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:28.087+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:28.087+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:28.087+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.000+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:29.116+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:29.116+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:29.116+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:29.117+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:29.129+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:29.129+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:29.129+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:29.130+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:28.969+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:30.084+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:30.084+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:30.084+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:30.084+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:30.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:30.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:30.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:30.084+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:29.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:29.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:29.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:29.960+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:31.058+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:31.058+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:31.058+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:31.058+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:31.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:31.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:31.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:31.131+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:30.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:30.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:30.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:30.985+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:32.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:31.724+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4898 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:32.075+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:32.075+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:32.075+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:32.075+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:32.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:32.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:32.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:32.152+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:32.009+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:32.009+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:32.009+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:32.009+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:33.273 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:33.099+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:33.273 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:33.099+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:33.273 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:33.099+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:33.099+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:33.154+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:33.154+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:33.154+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:33.154+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:33.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:33.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:33.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:33.008+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:33.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:33.008+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:34.091+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:34.091+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:34.091+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:34.092+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:34.165+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:34.165+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:34.165+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:34.165+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.014+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.014+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.014+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.014+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:35.087+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:35.087+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:35.087+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:35.087+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:35.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:35.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:35.122+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:35.123+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:34.983+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:36.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:36.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:36.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:36.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:36.006+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:36.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:36.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:36.113+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:36.113+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:36.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:36.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:36.124+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:36.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:36.124+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:37.039 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:36.725+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 4991 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:37.119+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:37.119+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:37.119+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:37.119+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:37.111+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:37.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:37.112+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:37.112+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:37.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:37.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:37.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:37.038+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:38.095+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:38.095+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:38.095+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:38.095+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:38.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:38.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:38.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:38.064+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.032+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:39.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:39.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:39.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:39.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:39.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:39.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:39.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:39.079+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.015+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:39.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:38.985+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:39.993+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:40.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:40.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:40.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:40.025+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:40.129+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:40.129+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:40.129+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:40.129+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:41.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:41.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:41.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:41.015+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:41.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:41.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:41.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:41.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:41.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:41.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:41.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:41.130+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:40.969+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:40.969+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:40.969+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:41.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:40.969+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:41.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:41.727+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5091 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:42.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:42.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:42.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:42.049+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:42.137+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:42.137+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:42.137+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:42.137+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:41.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:41.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:41.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:41.948+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:43.089+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:43.089+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:43.089+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:43.089+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:43.091+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:43.091+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:43.091+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:43.091+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:42.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:42.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:42.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:42.947+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:43.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:43.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:43.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:43.955+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:44.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:44.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:44.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:44.110+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:44.110+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:44.110+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:44.110+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:44.110+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:45.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:45.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:45.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:45.134+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:45.134+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:45.134+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:45.134+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:45.134+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:44.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:44.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:44.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:45.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:44.992+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:46.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:46.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:46.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:46.146+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:46.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:46.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:46.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:46.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:46.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:46.182+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:46.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:46.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:46.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:46.041+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:47.053 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:46.728+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5208 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:47.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:47.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:47.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:47.053+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:47.126+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:47.126+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:47.126+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:47.126+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:47.142+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:47.142+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:47.142+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:47.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:47.143+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.018+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.018+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.018+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.018+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:48.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:48.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:48.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:48.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:48.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:48.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:48.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:48.173+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:48.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:48.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:48.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:48.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:48.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:48.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:48.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:48.085+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:49.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:49.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:49.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:49.160+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:48.987+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:49.091+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:49.091+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:49.091+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:49.091+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:50.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:50.166+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:50.166+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:50.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:50.166+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:49.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:49.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:49.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:49.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:50.131+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:50.131+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:50.131+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:50.131+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:51.135+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:51.135+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:51.135+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:51.135+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:50.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:50.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:50.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:50.975+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:51.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:51.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:51.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:51.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:51.100+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:51.988 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:51.729+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5316 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:52.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:52.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:52.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:52.158+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:51.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:51.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:51.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:51.987+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:52.106+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:52.106+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:52.106+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:52.106+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:53.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:53.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:53.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:53.155+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:53.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:53.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:53.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:53.158+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:53.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:53.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:53.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:53.029+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:54.151+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:54.151+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:54.151+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:54.151+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:54.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:54.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:54.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:54.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:54.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:54.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:54.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:54.143+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:54.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:54.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:54.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.001+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:55.198+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:55.198+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:55.198+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:55.198+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:55.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:55.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:55.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:55.175+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:55.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:55.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:54.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:56.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:56.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:56.155+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:56.155+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:56.222+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:56.222+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:56.222+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:56.222+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:56.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:56.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:56.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:56.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:56.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:56.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:56.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:56.027+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:57.048 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:51:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:51:56.730+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5409 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:51:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:57.198+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:57.198+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:57.198+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:57.199+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:57.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:57.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:57.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:57.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:57.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:57.048+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:57.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:57.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:57.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:57.111+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:58.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:58.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:58.229+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:58.229+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:58.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:58.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:58.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:58.087+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:58.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:58.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:58.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:58.071+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:59.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:59.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:59.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:51:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:51:59.189+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:59.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:59.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:59.084+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:51:59.084+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:59.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:59.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:59.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:51:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:51:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:51:59.058+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:00.176+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:00.176+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:00.176+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:00.176+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:00.082+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:00.082+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:00.082+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:00.082+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:00.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:00.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:00.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:00.101+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:01.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:01.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:01.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:01.174+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:01.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:01.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:01.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:01.050+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:01.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:01.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:01.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:01.101+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:02.069 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:01.732+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5504 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:02.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:02.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:02.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:02.127+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:02.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:02.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:02.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:02.069+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:02.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:02.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:02.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:02.117+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:03.147+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:03.147+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:03.147+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:03.147+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:03.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:03.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:03.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:03.042+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:03.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:03.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:03.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:03.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:03.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:03.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:03.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:03.074+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:04.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:04.138+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:04.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:04.138+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:04.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:04.138+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:04.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:04.138+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:04.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:04.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:04.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:04.052+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:04.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:04.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:04.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:04.121+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:05.281 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:05.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:05.281 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:05.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:05.281 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:05.156+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:05.281 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:05.156+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:05.281 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:05.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:05.281 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:05.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:05.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:05.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:05.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:05.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:05.282 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:05.102+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:05.282 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:05.102+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:05.282 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:05.102+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:05.282 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:05.102+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:06.145+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:06.145+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:06.145+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:06.146+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:06.010+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:06.010+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:06.010+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:06.010+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:06.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:06.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:06.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:06.144+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:07.027 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:06.732+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5600 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:07.107+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:07.107+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:07.107+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:07.107+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:07.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:07.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:07.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:07.152+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:07.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:07.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:07.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:07.026+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:08.353 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:08.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:08.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:08.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:08.149+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:08.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:08.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:08.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:08.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:08.042+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:08.355 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:08.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:08.355 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:08.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:08.355 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:08.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:08.355 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:08.145+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:09.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:09.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:09.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:09.126+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.004+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:09.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:09.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:09.146+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:09.146+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:10.108+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:10.108+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:10.108+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:10.108+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:09.957+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:10.169+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:10.169+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:10.169+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:10.170+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:11.109+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:11.109+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:11.109+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:11.110+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:10.950+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:10.950+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:10.950+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:10.950+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:11.189+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:11.190+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:11.190+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:11.190+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:12.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:11.734+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5695 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:12.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:11.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:12.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:11.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:12.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:11.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:12.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:11.968+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:12.081+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:12.081+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:12.081+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:12.081+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:12.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:12.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:12.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:12.140+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:13.114+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:13.114+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:13.114+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:13.115+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:12.939+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:12.939+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:12.939+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:12.939+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:13.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:13.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:13.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:13.092+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:14.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:14.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:14.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:14.143+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:13.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:13.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:13.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:13.975+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:14.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:14.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:14.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:14.100+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:15.133+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:15.133+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:15.133+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:15.133+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:14.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:14.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:14.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:14.984+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:15.053+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:15.053+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:15.053+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:15.053+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:16.066+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:16.066+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:16.066+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:16.066+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:16.157+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:16.157+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:16.157+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:16.157+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:16.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:16.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:16.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:16.008+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:17.015 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:16.736+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5790 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:17.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:17.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:17.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:17.140+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:17.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:17.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:17.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:17.015+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.043+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:18.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:18.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:18.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:18.025+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:17.995+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:18.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:18.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:18.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:18.178+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:19.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:19.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:19.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:19.174+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.001+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.001+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:18.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:18.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:18.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:18.977+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:20.213 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:20.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:20.213 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:20.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:20.213 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:20.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:20.214 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:20.015+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:20.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.955+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:20.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.955+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:20.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.955+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:20.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:19.955+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:20.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:20.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:20.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:20.213+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:21.254 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:21.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:21.254 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:21.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:21.254 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:21.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:21.254 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:21.040+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:21.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:20.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:21.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:20.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:21.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:20.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:21.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:20.980+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:21.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:21.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:21.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:21.254+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:21.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:21.737+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5886 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:22.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:22.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:22.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:22.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:22.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:22.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:22.240 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:22.071+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:22.240 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:21.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:21.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:21.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:21.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:22.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:22.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:22.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:22.239+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:23.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:23.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:23.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:23.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:23.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:23.087+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:23.250 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:23.087+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:23.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:22.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:23.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:22.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:23.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:22.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:23.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:22.917+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:23.551 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:23.249+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:23.551 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:23.249+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:23.551 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:23.249+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:23.551 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:23.249+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:24.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:24.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:24.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:24.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:24.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:24.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:24.274 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:24.051+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:24.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:23.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:24.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:23.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:24.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:23.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:24.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:23.874+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:24.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:24.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:24.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:24.273+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:25.243 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:25.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:25.243 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:25.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:25.243 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:25.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:25.243 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:25.052+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:25.244 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:24.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:25.244 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:24.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:25.244 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:24.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:25.244 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:24.883+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:25.242+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:25.242+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:25.242+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:25.242+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:26.259 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:26.093+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:26.093+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:26.093+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:26.093+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:25.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:25.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:25.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:26.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:25.874+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:26.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:26.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:26.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:26.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:27.056 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:26.738+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 5982 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:27.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:26.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:27.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:26.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:27.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:26.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:27.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:26.885+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:27.055+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:27.055+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:27.055+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:27.055+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:27.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:27.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:27.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:27.304+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:28.092 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:27.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:28.092 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:27.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:28.092 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:27.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:28.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:27.837+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:28.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:28.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:28.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:28.261+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:28.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:28.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:28.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:28.092+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:28.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:28.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:28.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:28.850+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:29.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:29.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:29.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:29.058+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:29.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:29.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:29.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:29.235+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:29.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:29.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:29.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:29.858+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:30.102+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:30.102+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:30.102+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:30.102+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:30.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:30.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:30.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:30.201+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:30.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:30.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:30.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:30.818+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:31.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:31.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:31.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:31.228+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:31.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:31.150+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:31.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:31.150+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:31.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:31.150+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:31.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:31.150+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:31.739+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6077 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:31.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:31.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:31.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:31.803+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:32.263+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:32.263+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:32.263+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:32.263+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:32.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:32.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:32.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:32.164+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:33.119 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:32.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:33.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:32.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:33.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:32.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:33.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:32.853+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:33.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:33.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:33.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:33.253+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:33.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:33.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:33.118+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:33.119+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:33.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:33.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:33.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:33.815+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:34.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:34.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:34.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:34.145+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:34.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:34.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:34.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:34.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:34.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:34.256+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:34.807+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:34.807+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:34.807+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:34.807+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:35.141+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:35.141+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:35.141+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:35.141+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:35.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:35.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:35.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:35.264+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:35.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:35.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:35.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:35.806+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:36.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:36.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:36.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:36.164+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:36.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:36.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:36.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:36.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:36.269+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:36.741+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6172 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:36.849+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:36.849+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:36.849+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:36.849+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:37.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:37.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:37.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:37.236+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:37.165+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:37.165+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:37.165+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:37.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:37.165+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:37.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:37.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:37.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:37.819+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:38.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:38.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:38.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:38.274+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:38.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:38.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:38.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:38.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:38.160+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:38.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:38.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:38.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:38.819+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:39.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:39.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:39.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:39.236+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:39.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:39.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:39.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:39.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:39.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:39.140+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:39.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:39.140+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:39.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:39.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:39.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:39.802+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:40.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:40.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:40.236+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:40.236+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:40.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:40.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:40.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:40.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:40.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:40.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:40.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:40.166+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:40.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:40.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:40.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:40.773+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:41.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:41.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:41.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:41.277+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:41.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:41.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:41.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:41.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:41.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:41.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:41.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:41.122+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:41.742+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6268 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:41.745+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:41.745+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:41.745+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:41.745+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:42.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:42.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:42.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:42.239+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:42.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:42.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:42.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:42.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:42.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:42.143+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:42.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:42.143+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:42.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:42.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:42.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:42.716+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:43.551 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:43.267+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:43.267+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:43.267+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:43.267+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:43.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:43.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:43.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:43.552 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:43.124+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:43.666+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:43.666+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:43.666+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:43.666+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:44.158+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:44.158+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:44.158+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:44.158+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:44.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:44.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:44.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:44.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:44.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:44.270+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:44.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:44.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:44.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:44.681+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:45.129+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:45.129+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:45.129+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:45.129+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:45.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:45.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:45.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:45.291+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:45.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:45.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:45.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:45.702+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:46.127+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:46.127+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:46.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:46.128+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:46.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:46.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:46.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:46.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:46.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:46.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:46.331+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:47.105 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:46.742+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6364 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:47.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:46.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:47.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:46.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:47.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:46.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:47.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:46.662+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:47.305+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:47.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:47.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:47.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:47.104+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:48.072 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:47.664+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:48.072 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:47.664+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:48.072 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:47.664+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:48.072 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:47.664+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:48.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:48.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:48.071+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:48.071+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:48.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:48.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:48.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:48.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:48.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:48.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:48.305+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:49.030 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:48.646+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:49.030 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:48.646+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:49.030 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:48.646+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:49.030 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:48.646+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:49.029+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:49.029+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:49.029+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:49.029+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:49.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:49.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:49.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:49.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:49.265+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:50.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:49.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:50.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:49.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:50.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:49.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:50.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:49.687+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:50.301 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:50.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:50.302 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:50.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:50.302 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:50.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:50.302 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:50.007+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:50.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:50.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:50.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:50.301+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:51.052 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:50.708+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:51.052 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:50.708+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:51.052 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:50.708+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:51.052 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:50.708+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:51.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:51.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:51.051+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:51.051+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:51.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:51.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:51.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:51.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:51.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:51.276+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:52.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:51.745+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6459 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:52.064 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:51.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:52.064 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:51.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:52.064 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:51.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:52.065 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:51.725+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:52.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:52.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:52.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:52.284+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:52.063+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:52.063+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:52.063+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:52.063+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:53.034 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:52.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:53.035 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:52.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:53.035 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:52.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:53.035 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:52.746+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:53.034+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:53.034+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:53.034+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:53.034+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:53.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:53.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:53.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:53.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:53.283+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:53.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:53.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:53.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:53.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:54.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:54.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:54.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:54.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:54.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:54.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:54.296 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:54.032+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:54.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:54.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:54.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:54.295+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:55.076 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:54.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:55.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:54.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:55.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:54.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:55.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:54.730+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:55.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:55.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:55.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:55.076+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:55.292+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:55.292+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:55.292+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:55.292+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:56.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:55.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:56.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:55.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:56.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:55.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:56.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:55.751+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:56.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:56.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:56.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:56.037+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:56.266+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:56.266+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:56.266+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:56.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:56.266+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:56.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-02-10T06:52:56.923+0000 7f7060b5d700 0 monclient(hunting): authenticate timed out after 300 2024-02-10T06:52:56.926 INFO:teuthology.orchestra.run.smithi073.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T06:52:57.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:52:56.746+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6553 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:52:57.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:56.747+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:57.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:56.747+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:57.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:56.747+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:57.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:56.747+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:57.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:57.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:57.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:57.081+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:57.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:57.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:57.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:57.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:57.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:52:57.669 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps'" 2024-02-10T06:52:57.884 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8d65087d064143e1b5050021729893d2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps'" 2024-02-10T06:52:57.887 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-10T06:52:57.935 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-10T06:52:57.935 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:52:57.935 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:52:57.975 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:52:57.975 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:52:58.034 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-10T06:52:58.070 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:57.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:58.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:57.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:58.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:57.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:58.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:57.750+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:58.262+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:58.262+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:58.262+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:58.262+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:58.070+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:58.070+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:58.070+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:58.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:58.070+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:59.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:58.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:59.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:58.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:59.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:58.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:52:59.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:58.706+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:59.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:59.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:59.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:52:59.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:52:59.214+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:52:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:59.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:59.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:59.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:52:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:52:59.062+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:00.095 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:59.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:00.095 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:59.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:00.095 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:59.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:00.095 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:52:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:52:59.722+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:00.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:00.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:00.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:00.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:00.207+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:00.094+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:00.094+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:00.094+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:00.094+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:00.747+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:00.747+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:00.747+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:00.747+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:01.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:01.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:01.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:01.074+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:01.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:01.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:01.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:01.244+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:02.044 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:01.748+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6650 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:02.044 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:01.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:02.044 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:01.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:02.044 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:01.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:02.044 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:01.710+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:02.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:02.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:02.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:02.268+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:02.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:02.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:02.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:02.043+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:03.007 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:02.679+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:03.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:02.679+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:03.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:02.679+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:03.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:02.679+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:03.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:03.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:03.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:03.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.007+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:03.288+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:03.288+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:03.288+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:03.288+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:03.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:03.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:03.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:03.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:03.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:03.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:03.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:03.727+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:04.279 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:04.279 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:04.279 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:04.279 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:03.993+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:04.278+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:04.278+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:04.278+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:04.278+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:05.039 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:04.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:05.039 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:04.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:05.039 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:04.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:05.040 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:04.744+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:05.315 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:05.039+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:05.315 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:05.039+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:05.315 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:05.039+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:05.315 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:05.039+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:05.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:05.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:05.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:05.314+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:06.031 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:05.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:06.032 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:05.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:06.032 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:05.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:06.032 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:05.751+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:06.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:06.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:06.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:06.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:06.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:06.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:06.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:06.031+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:06.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:06.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:06.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:06.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:07.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:06.749+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6745 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:07.077 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:06.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:39.348088+0000 front 2024-02-10T06:42:39.348056+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:07.078 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:06.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:39.348072+0000 front 2024-02-10T06:42:39.348142+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:07.078 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:06.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:39.348118+0000 front 2024-02-10T06:42:39.348103+0000 (oldest deadline 2024-02-10T06:43:02.847943+0000) 2024-02-10T06:53:07.078 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:06.739+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:07.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:07.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:07.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:07.077+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:07.711 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:07.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:07.711 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:07.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:07.711 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:07.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:07.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:07.404+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:08.088 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:07.710+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:08.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:08.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:08.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:08.088+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:08.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:08.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:08.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:08.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:08.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:08.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:08.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:08.404+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:09.104 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:08.677+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:09.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:09.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:09.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:09.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:09.104+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:09.104+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:09.370+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:09.370+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:09.370+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:09.370+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:10.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:09.686+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:10.089+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:10.089+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:10.089+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:10.089+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:10.735 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:10.380+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:10.735 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:10.380+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:10.736 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:10.380+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:10.736 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:10.380+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:11.041 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:10.734+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:11.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:11.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:11.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:11.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:11.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:11.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:11.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:11.041+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:11.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:11.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:11.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:11.367+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:12.086 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:11.750+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6841 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:12.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:11.745+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:12.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:12.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:12.085+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:12.085+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:12.708 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:12.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:12.708 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:12.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:12.708 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:12.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:12.709 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:12.400+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:13.041 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:12.707+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.040+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:13.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:13.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:13.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:13.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:13.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:13.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:13.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:13.424+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:13.996 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:13.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:13.996+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:14.727 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:14.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:14.728 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:14.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:14.728 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:14.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:14.728 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:14.402+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:14.727+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:14.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:14.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:15.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:14.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:15.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:14.971+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:15.357+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:15.357+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:15.357+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:15.357+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:15.770+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:15.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:38.245923+0000 front 2024-02-10T06:42:38.245930+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:15.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:38.245962+0000 front 2024-02-10T06:42:38.245972+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:15.995+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:38.245954+0000 front 2024-02-10T06:42:38.245985+0000 (oldest deadline 2024-02-10T06:43:02.345798+0000) 2024-02-10T06:53:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:15.995+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:16.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:16.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:16.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:16.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:16.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:16.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:16.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:16.320+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:17.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:16.767+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:17.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:16.752+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 6935 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:17.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:17.041+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:17.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:17.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:17.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:17.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:17.320+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:18.052 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:17.756+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:18.304 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:18.052+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:18.303+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:18.303+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:18.303+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:18.303+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:19.009 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:18.736+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:19.349 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:19.095+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:19.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:19.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:19.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:19.349+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:19.714+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:20.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:20.143+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:20.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 since back 2024-02-10T06:42:40.382097+0000 front 2024-02-10T06:42:40.382011+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:20.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 since back 2024-02-10T06:42:40.381998+0000 front 2024-02-10T06:42:40.382030+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:20.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 since back 2024-02-10T06:42:40.382120+0000 front 2024-02-10T06:42:40.382055+0000 (oldest deadline 2024-02-10T06:43:04.482003+0000) 2024-02-10T06:53:20.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:20.321+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:21.119 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:20.688+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:21.119+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:21.342+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:22.079 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:21.704+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:22.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:21.753+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7032 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:22.313+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:22.079+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:23.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:22.728+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:23.327+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:23.092+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:23.728+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:24.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:24.079+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:24.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:25.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:24.747+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:25.412 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:25.120+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:25.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:25.411+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:25.699+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:26.409 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:26.146+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:26.695 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:26.408+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:26.753+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7127 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:26.694+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:27.393+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:27.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:27.184+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:27.722+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:28.358+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:28.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:28.219+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:28.708+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:29.387+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:29.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:29.246+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:29.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:30.388+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:30.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:30.278+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:30.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:30.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:30.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:30.769+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:31.420+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:31.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:31.286+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:31.756+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7223 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:31.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:31.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:31.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:31.768+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:32.391+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:32.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:32.266+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:32.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:32.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:32.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:32.802+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:33.555 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:33.354+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:33.556 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:33.232+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:33.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:33.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:33.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:33.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:33.801+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:34.321+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:34.183+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:34.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:34.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:34.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:34.840+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:35.317+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:35.140+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:35.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:35.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:35.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:35.792+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:36.308+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:36.183+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:36.757+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7317 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:36.745+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:36.745+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:36.745+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:36.745+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:37.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:37.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:37.145+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:37.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:37.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:37.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:37.745+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:38.293+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:38.137+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:39.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:38.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:39.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:38.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:39.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:38.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:39.091 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:38.748+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:39.275+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:39.089+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:40.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:39.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:40.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:39.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:40.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:39.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:40.043 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:39.755+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:40.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:40.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:40.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:40.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:40.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:40.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:40.327 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:40.041+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:40.323+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:41.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:40.766+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:41.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:40.766+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:41.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:40.766+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:41.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:40.766+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:41.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:41.663 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:41.373+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:41.981 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:41.759+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7414 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:41.981 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:41.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:41.982 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:41.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:41.982 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:41.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:41.982 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:41.762+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:42.348 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:42.349 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:42.349 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:42.349 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:41.981+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:42.348+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:42.348+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:42.348+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:42.348+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:42.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:42.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:42.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:42.768+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:42.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:42.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:42.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:42.972+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:43.585 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:43.315+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:43.585 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:43.315+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:43.585 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:43.315+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:43.585 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:43.315+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:43.760+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:43.760+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:43.760+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:43.760+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:44.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:43.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:44.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:43.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:44.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:43.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:44.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:43.944+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:44.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:44.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:44.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:44.274+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:44.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:44.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:44.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:44.991+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:44.793+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:44.793+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:44.793+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:44.793+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:45.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:45.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:45.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:45.271+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:45.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:45.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:45.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:45.755+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:45.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:45.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:45.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:45.994+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:46.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:46.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:46.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:46.225+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:46.760+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7509 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:46.757+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:46.757+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:46.757+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:46.757+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:46.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:46.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:46.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:46.948+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:47.231+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:47.231+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:47.231+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:47.231+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:47.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:47.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:47.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:47.746+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:47.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:47.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:47.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:47.972+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:48.251+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:48.251+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:48.251+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:48.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:48.251+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:48.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:48.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:48.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:48.946+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:49.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:48.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:48.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:48.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:48.795+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:49.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:49.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:49.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:49.261+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:49.897+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:49.897+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:49.897+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:49.897+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:49.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:49.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:49.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:49.828+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:50.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:50.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:50.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:50.310+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:50.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:50.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:50.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:50.864+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:50.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:50.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:50.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:50.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:51.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:51.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:51.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:51.322+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:51.934 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:51.762+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7605 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:51.934 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:51.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:51.934 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:51.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:51.934 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:51.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:51.934 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:51.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:52.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:51.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:52.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:51.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:52.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:51.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:52.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:51.933+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:52.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:52.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:52.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:52.319+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:52.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:52.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:52.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:52.968+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:52.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:52.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:52.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:52.909+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:53.557 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:53.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:53.557 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:53.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:53.558 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:53.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:53.558 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:53.305+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:54.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:53.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:54.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:53.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:54.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:53.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:54.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:53.930+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:54.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:54.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:54.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:54.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:54.275 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:54.007+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:54.276 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:54.007+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:54.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:54.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:54.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:54.274+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:55.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:54.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:55.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:54.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:55.308 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:54.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:55.309 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:54.965+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:55.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.010+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:55.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.010+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:55.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.010+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:55.310 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.010+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:55.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:55.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:55.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:55.308+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:56.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:55.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:56.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:55.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:56.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:55.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:56.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:55.924+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:56.312 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:56.312 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:56.312 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:56.312 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:55.974+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:56.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:56.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:56.310+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:56.310+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:57.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:53:56.764+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7699 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:53:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:56.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:56.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:56.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:56.919+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:57.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:56.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:57.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:56.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:57.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:56.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:57.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:56.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:57.306+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:57.306+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:57.306+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:57.306+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:58.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:57.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:58.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:57.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:58.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:57.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:58.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:57.978+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:58.324 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:57.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:58.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:57.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:58.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:57.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:58.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:57.938+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:58.323+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:58.323+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:58.323+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:58.323+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:59.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:58.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:59.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:58.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:59.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:58.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:53:59.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:58.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:59.278 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:58.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:59.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:58.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:59.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:58.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:53:59.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:58.897+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:53:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:59.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:59.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:59.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:53:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:53:59.277+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:00.284 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:59.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:00.284 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:59.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:00.284 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:59.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:00.284 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:53:59.917+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:00.285 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:59.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:00.285 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:59.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:00.285 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:59.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:00.285 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:53:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:53:59.894+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:00.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:00.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:00.284+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:00.284+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:01.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:00.910+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:01.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:00.910+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:01.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:00.910+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:01.278 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:00.910+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:01.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:00.871+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:01.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:00.871+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:01.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:00.871+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:01.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:00.871+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:01.278+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:01.278+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:01.278+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:01.278+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:01.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:01.765+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7796 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:01.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:01.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:01.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:01.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:01.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:01.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:01.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:01.889+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:02.247 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:01.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:02.247 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:01.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:02.247 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:01.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:02.247 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:01.915+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:02.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:02.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:02.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:02.247+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:03.203 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:02.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:03.204 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:02.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:03.204 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:02.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:03.204 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:02.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:03.204 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:02.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:03.204 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:02.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:03.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:02.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:03.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:02.903+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:03.558 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:03.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:03.558 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:03.202+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:03.558 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:03.202+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:03.558 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:03.202+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:03.859+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:03.859+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:03.859+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:03.860+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:03.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:03.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:03.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:03.854+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:04.176+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:04.176+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:04.176+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:04.176+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:05.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:04.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:05.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:04.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:05.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:04.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:05.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:04.824+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:05.089 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:04.813+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:05.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:04.813+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:05.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:04.813+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:05.090 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:04.813+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:05.198+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:05.198+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:05.198+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:05.198+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:05.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:05.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:05.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:05.846+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:05.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:05.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:05.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:05.815+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:06.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:06.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:06.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:06.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:07.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:06.765+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7891 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:06.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:06.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:06.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:06.844+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:06.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:06.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:06.790+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:06.790+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:07.238+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:07.238+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:07.238+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:07.238+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:07.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:07.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:07.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:07.847+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:07.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:07.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:07.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:07.836+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:08.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:08.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:08.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:08.193+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:08.826+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:08.826+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:08.826+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:08.826+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:08.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:08.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:08.824+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:08.824+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:09.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:09.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:09.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:09.233+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:09.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:09.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:09.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:09.838+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:09.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:10.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:09.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:10.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:09.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:10.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:09.872+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:10.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:10.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:10.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:10.283+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:10.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:10.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:10.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:10.857+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:10.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:10.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:10.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:10.880+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:11.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:11.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:11.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:11.302+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:12.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:11.768+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 7987 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:11.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:11.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:11.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:11.899+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:11.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:11.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:11.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:11.896+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:12.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:12.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:12.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:12.349+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:13.365 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:12.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:13.366 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:12.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:13.366 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:12.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:13.366 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:12.899+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:13.366 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:12.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:13.366 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:12.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:13.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:12.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:13.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:12.932+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:13.364+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:13.364+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:13.364+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:13.365+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:13.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:13.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:13.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:13.865+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:13.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:13.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:13.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:13.913+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:14.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:14.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:14.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:14.390+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:14.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:15.157 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:14.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:15.157 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:14.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:15.157 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:14.823+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:15.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:14.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:15.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:14.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:15.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:14.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:15.158 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:14.937+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:15.787 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:15.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:15.787 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:15.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:15.787 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:15.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:15.787 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:15.391+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:15.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:15.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:15.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:15.786+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:15.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:15.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:15.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:15.960+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:16.770 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:16.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:16.771 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:16.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:16.771 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:16.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:16.771 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:16.438+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:16.769+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8081 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:16.824+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:16.824+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:16.824+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:17.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:16.824+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:16.949+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:16.949+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:16.949+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:16.949+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:17.861 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:17.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:17.861 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:17.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:17.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:17.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:17.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:17.399+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:17.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:17.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:17.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:17.861+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:17.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:17.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:17.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:17.925+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:18.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:18.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:18.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:18.435+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:18.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:18.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:18.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:18.894+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:18.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:18.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:18.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:18.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:19.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:19.460+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:19.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:19.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:19.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:19.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:19.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:19.461+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:19.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:19.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:19.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:19.865+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:19.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:19.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:19.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:19.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:20.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:20.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:20.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:20.507+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:20.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:20.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:20.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:20.889+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:20.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:20.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:20.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:20.885+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:21.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:21.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:21.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:21.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:21.471+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:21.771+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8178 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:21.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:21.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:21.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:21.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:21.863+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:21.863+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:21.863+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:21.863+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:22.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:22.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:22.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:22.458+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:22.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:22.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:22.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:22.878+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:22.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:22.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:22.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:22.914+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:23.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:23.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:23.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:23.429+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:23.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:23.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:23.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:23.882+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:23.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:23.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:23.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:23.930+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:24.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:24.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:24.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:24.420+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:24.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:24.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:24.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:24.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:24.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:24.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:24.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:24.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:25.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:25.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:25.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:25.417+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:25.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:25.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:25.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:25.913+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:25.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:25.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:25.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:25.948+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:26.773 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:26.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:26.774 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:26.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:26.774 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:26.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:26.774 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:26.417+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:26.772+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8273 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:26.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:26.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:26.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:26.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:26.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:26.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:27.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:26.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:27.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:26.988+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:27.464+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:27.464+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:27.464+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:27.464+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:27.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:27.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:27.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:27.993+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:27.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:27.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:27.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:28.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:27.903+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:28.437+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:28.437+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:28.437+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:28.437+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.001+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:28.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:29.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:28.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:29.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:28.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:29.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:28.937+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:29.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:29.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:29.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:29.432+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:29.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:29.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:29.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:29.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:29.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:30.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:30.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:30.399+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:30.399+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:31.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:31.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:31.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:31.034+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:30.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:30.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:30.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:30.935+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:31.666 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:31.398+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:31.667 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:31.398+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:31.667 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:31.398+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:31.667 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:31.398+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:31.934 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:31.774+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8369 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:32.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:32.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:32.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:32.064+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:31.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:31.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:31.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:31.933+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:32.737 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:32.397+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:32.737 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:32.397+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:32.737 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:32.397+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:32.737 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:32.397+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:33.068 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:32.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:33.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:32.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:33.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:32.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:33.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:32.914+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:33.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:33.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:33.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:33.068+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:33.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:33.372+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:33.372+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:33.372+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:34.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:34.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:34.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:34.044+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:33.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:33.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:33.933+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:33.933+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:34.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:34.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:34.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:34.390+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:35.350 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:35.350 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:35.350 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:35.350 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.027+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:35.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:34.908+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:35.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:34.908+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:35.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:34.908+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:35.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:34.908+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:35.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:35.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:35.349+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:35.349+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:35.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:35.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:35.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:35.917+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:35.980+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:36.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:36.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:36.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:36.308+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:36.775+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8463 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:36.979+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:36.979+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:36.979+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:36.979+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:36.908+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:36.908+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:36.908+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:36.908+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:37.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:37.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:37.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:37.331+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:38.332 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:37.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:38.332 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:37.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:38.332 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:37.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:38.332 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:37.893+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:38.333 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:37.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:38.333 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:37.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:38.333 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:37.970+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:38.333 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:37.970+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:38.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:38.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:38.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:38.331+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:39.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:38.926+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:39.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:38.926+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:39.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:38.926+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:39.337 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:38.926+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:39.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:38.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:39.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:38.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:39.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:38.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:39.337 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:38.972+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:39.336+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:39.336+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:39.336+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:39.336+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:40.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:39.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:40.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:39.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:40.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:39.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:40.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:39.950+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:40.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:39.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:40.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:39.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:40.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:39.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:40.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:39.954+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:40.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:40.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:40.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:40.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:40.334+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:41.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:40.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:41.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:40.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:41.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:40.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:41.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:40.916+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:41.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:40.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:41.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:40.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:41.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:40.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:41.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:40.998+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:41.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:41.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:41.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:41.319+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:41.953 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:41.777+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8560 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:41.953 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:41.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:41.953 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:41.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:41.953 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:41.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:41.954 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:41.880+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:42.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:41.999+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:42.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:41.999+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:42.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:41.999+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:42.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:41.999+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:42.325+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:42.325+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:42.325+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:42.325+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:43.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:42.978+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:43.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:42.978+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:43.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:42.978+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:43.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:42.978+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:43.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:42.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:43.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:42.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:43.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:42.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:43.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:42.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:43.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:43.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:43.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:43.314+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:44.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:43.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:44.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:43.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:44.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:43.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:44.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:43.981+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:44.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:43.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:44.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:43.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:44.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:43.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:44.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:43.883+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:44.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:44.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:44.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:44.319+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:45.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:44.966+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:45.276 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:44.966+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:45.276 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:44.966+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:45.276 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:44.966+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:45.276 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:44.875+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:45.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:44.875+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:45.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:44.875+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:45.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:44.875+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:45.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:45.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:45.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:45.275+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:46.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:45.979+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:46.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:45.979+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:46.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:45.979+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:46.283 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:45.979+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:46.283 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:45.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:46.283 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:45.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:46.283 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:45.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:46.283 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:45.876+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:46.280+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:46.280+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:46.280+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:46.280+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:46.777+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8655 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.023+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:46.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:46.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:46.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:46.828+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:47.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:47.305+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:47.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:47.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:47.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:47.865+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:47.987+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:48.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:48.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:48.295+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:48.295+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:48.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:48.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:48.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:48.847+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:48.995+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:48.995+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:48.995+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:48.996+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:49.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:49.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:49.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:49.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:49.333+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:50.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:49.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:50.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:49.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:50.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:49.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:50.322 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:49.968+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:50.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:49.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:50.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:49.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:50.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:49.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:50.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:49.882+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:50.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:50.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:50.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:50.320+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:51.357 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:51.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:51.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:51.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:51.005+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:50.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:50.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:50.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:51.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:50.893+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:51.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:51.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:51.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:51.357+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:52.020 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:51.780+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8750 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:52.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:51.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:52.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:51.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:52.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:51.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:52.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:51.847+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:52.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:52.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:52.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:52.020+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:52.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:52.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:52.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:52.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:52.371+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:53.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:53.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:53.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:53.007+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:52.890+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:52.890+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:52.890+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:52.890+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:53.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:53.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:53.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:53.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:53.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:53.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:53.849 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:53.418+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:54.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:54.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:54.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:54.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:54.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:54.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:54.030+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:54.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:53.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:54.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:53.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:54.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:53.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:54.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:53.847+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:54.372+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:54.373+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:54.373+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:54.373+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:55.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:55.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:55.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:55.076+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:54.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:54.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:54.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:54.893+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:55.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:55.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:55.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:55.395+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:56.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:56.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:56.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:56.086+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:55.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:55.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:55.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:55.881+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:56.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:56.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:56.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:56.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:56.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:56.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:56.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:56.393+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:57.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:54:56.782+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8843 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:54:57.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:56.853+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:57.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:56.853+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:57.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:56.853+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:57.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:56.853+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:57.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:57.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:57.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:57.119+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:57.350+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:57.350+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:57.350+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:57.350+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-10T06:54:58.111 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-10T06:54:58.111 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:54:58.111 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:54:58.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:57.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:57.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:57.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:57.860+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:58.132 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-02-10T06:54:58.132 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:54:58.190 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-10T06:54:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:58.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:58.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:58.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:58.352+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:58.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:58.125+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:58.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:58.125+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:58.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:58.125+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:58.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:58.126+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:59.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:58.841+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:59.095 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:58.841+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:59.095 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:58.841+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:54:59.095 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:58.841+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:59.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:59.094+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:59.348 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:59.094+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:59.348 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:59.094+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:54:59.348 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:54:59.094+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:54:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:59.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:59.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:59.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:54:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:54:59.347+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:00.049+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:00.049+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:00.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:00.050+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:59.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:59.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:59.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:54:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:54:59.860+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:00.869 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:00.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:00.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:00.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:00.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:00.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:00.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:00.379+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:00.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:00.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:00.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:00.869+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:01.078+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:01.078+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:01.078+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:01.079+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:01.668 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:01.398+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:01.669 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:01.398+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:01.669 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:01.398+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:01.669 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:01.398+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:02.108 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:01.783+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 8939 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:02.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:01.910+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:02.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:01.910+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:02.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:01.910+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:02.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:01.910+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:02.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:02.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:02.107+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:02.107+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:02.445+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:02.445+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:02.445+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:02.445+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:02.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:02.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:02.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:02.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:03.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:03.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:03.098+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:03.099+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:03.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:03.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:03.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:03.412+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:03.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:03.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:03.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:03.979+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:04.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:04.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:04.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:04.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:04.109+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:04.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:04.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:04.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:04.432+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:04.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:04.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:04.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:04.988+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:05.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:05.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:05.085+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:05.085+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:05.443+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:05.443+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:05.443+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:05.443+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:06.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:06.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:06.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:06.037+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:06.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:05.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:05.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:05.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:05.975+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:06.786 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:06.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:06.786 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:06.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:06.786 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:06.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:06.786 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:06.484+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:07.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:06.785+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9033 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:07.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:06.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:07.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:06.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:07.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:06.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:07.065 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:06.955+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:07.063+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:07.063+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:07.064+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:07.064+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:07.451+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:07.451+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:07.451+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:07.451+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:07.926+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:07.926+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:07.926+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:07.926+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:08.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:08.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:08.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:08.045+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:08.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:08.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:08.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:08.485+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:08.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:08.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:08.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:08.887+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.044+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:09.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:09.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:09.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:09.440+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:09.894+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:09.894+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:09.894+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:09.894+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:10.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:10.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:10.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:10.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:09.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:10.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:10.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:10.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:10.438+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:10.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:10.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:10.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:10.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:11.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:11.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:11.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:11.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:11.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:11.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:11.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:11.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:11.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:11.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:11.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:11.458+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:12.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:11.787+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9125 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:12.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:12.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:12.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:12.031+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:11.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:11.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:11.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:11.990+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:12.672 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:12.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:12.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:12.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:12.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:12.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:12.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:12.410+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:12.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:12.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:12.994+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:12.994+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:13.061+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:13.061+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:13.061+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:13.061+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:13.388+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:13.388+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:13.388+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:13.388+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:13.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:13.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:13.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:13.955+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:14.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:14.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:14.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:14.423+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:14.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:14.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:14.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:14.928+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:14.980+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:15.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:15.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:15.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:15.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:15.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:15.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:15.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:15.374+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:16.017 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:15.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:16.018 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:15.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:16.018 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:15.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:16.018 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:15.951+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:16.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:16.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:16.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:16.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:16.789 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:16.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:16.790 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:16.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:16.790 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:16.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:16.790 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:16.372+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:16.788+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9215 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:16.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:16.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:16.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:16.944+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:17.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.003+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:17.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.003+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:17.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.003+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:17.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.003+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:17.347+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:17.347+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:17.347+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:17.347+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:18.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:18.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:18.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:18.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:17.976+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:18.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:17.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:18.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:17.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:18.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:17.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:18.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:17.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:18.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:18.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:18.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:18.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:19.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:19.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:19.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:19.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.023+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:19.325 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:18.907+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:19.326 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:18.907+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:19.326 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:18.907+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:19.326 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:18.907+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:19.324+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:19.324+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:19.324+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:19.324+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:20.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:20.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:20.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:20.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:19.989+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:20.329 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:19.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:20.329 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:19.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:20.329 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:19.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:20.329 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:19.902+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:20.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:20.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:20.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:20.327+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:21.327 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:20.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:20.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:20.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:20.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:20.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:20.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:20.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:21.328 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:20.899+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:21.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:21.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:21.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:21.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:21.625 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:21.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:21.625 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:21.327+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:21.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:21.790+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9303 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:22.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:21.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:22.357 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:21.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:22.357 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:21.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:22.357 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:21.931+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:22.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:21.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:22.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:21.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:22.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:21.935+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:22.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:21.935+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:22.355+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:22.355+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:22.356+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:22.356+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:22.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:22.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:22.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:22.916+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:22.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:22.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:22.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:22.920+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:23.376+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:23.376+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:23.376+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:23.377+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:23.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:23.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:23.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:23.883+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:23.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:23.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:23.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:23.962+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:24.422+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:24.422+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:24.422+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:24.422+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:24.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:24.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:24.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:24.930+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:24.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:24.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:24.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:24.956+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:25.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:25.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:25.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:25.428+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:25.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:25.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:25.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:25.942+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:25.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:25.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:25.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:25.915+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:26.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:26.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:26.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:26.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:26.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:26.395+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:26.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:26.395+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:26.791+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9388 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:26.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:26.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:26.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:26.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:26.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:26.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:26.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:26.930+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:27.364+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:27.364+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:27.364+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:27.364+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:28.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:27.933+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:28.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:27.933+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:28.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:27.933+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:28.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:27.933+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:28.331 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:27.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:28.331 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:27.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:28.331 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:27.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:28.331 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:27.920+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:28.329+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:28.329+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:28.329+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:28.329+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:29.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:28.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:29.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:28.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:29.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:28.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:29.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:28.915+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:29.346 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:28.890+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:29.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:28.890+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:29.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:28.890+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:29.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:28.890+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:29.345+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:29.345+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:29.345+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:29.345+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:30.317 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:29.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:30.317 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:29.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:30.318 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:29.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:30.318 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:29.930+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:30.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:29.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:30.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:29.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:30.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:29.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:30.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:29.961+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:30.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:30.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:30.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:30.317+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:31.345 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:30.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:31.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:30.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:31.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:30.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:31.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:30.959+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:31.346 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:30.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:31.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:30.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:31.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:30.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:31.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:30.883+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:31.345+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:31.345+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:31.345+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:31.345+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:31.972 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:31.794+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9477 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:31.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:31.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:31.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:31.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:31.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:31.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:31.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:31.893+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:32.317 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:31.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:32.317 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:31.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:32.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:31.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:32.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:31.971+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:32.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:32.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:32.316+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:32.317+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:33.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:33.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:33.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:33.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:33.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:33.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:33.320 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:33.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:33.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:32.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:33.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:32.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:33.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:32.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:33.320 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:32.881+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:33.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:33.318+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:33.318+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:33.318+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:33.319+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:34.177 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:33.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:34.177 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:33.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:34.177 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:33.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:34.177 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:33.916+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:34.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.028+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:34.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:34.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.029+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:34.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.029+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:34.321+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:34.321+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:34.321+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:34.321+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:35.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:35.290 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:35.290 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:35.290 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:34.981+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:35.290 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:34.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:35.291 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:34.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:35.291 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:34.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:35.291 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:34.959+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:35.289+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:35.289+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:35.289+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:35.289+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:36.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:35.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:36.316 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:35.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:36.316 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:35.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:36.316 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:35.948+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:36.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:35.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:36.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:35.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:36.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:35.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:36.317 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:35.936+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:36.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:36.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:36.314+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:36.314+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:36.795+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9565 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:36.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:36.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:36.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:36.994+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:36.897+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:36.897+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:36.897+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:36.897+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:37.377 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:37.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:37.378 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:37.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:37.378 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:37.343+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:37.378 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:37.344+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:38.359 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:38.359 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:38.359 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:38.359 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.005+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:38.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:37.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:38.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:37.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:38.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:37.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:38.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:37.912+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:38.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:38.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:38.358+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:38.358+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:38.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:38.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:38.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:38.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:38.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:39.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:39.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:39.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:39.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:39.401+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:39.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:39.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:39.954+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:39.954+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:39.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:39.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:39.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:40.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:39.922+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:40.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:40.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:40.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:40.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:40.380+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:40.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:40.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:40.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:40.991+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:40.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:40.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:40.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:40.971+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:41.672 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:41.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:41.672 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:41.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:41.672 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:41.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:41.672 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:41.391+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:41.922 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:41.797+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9658 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:41.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:41.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:41.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:41.981+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:41.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:41.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:41.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:41.985+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:42.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:42.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:42.391+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:42.391+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:43.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:42.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:43.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:42.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:42.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:42.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:42.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:42.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:42.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:42.940+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:43.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:43.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:43.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:43.369+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:43.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:43.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:43.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:43.971+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.000+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:44.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:44.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:44.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:44.405+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:44.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:44.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:44.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:44.956+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:44.951+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:45.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:45.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:45.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:45.431+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:45.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:45.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:45.903+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:45.903+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:45.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:45.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:45.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:45.944+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:46.799 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:46.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:46.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:46.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:46.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:46.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:46.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:46.435+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:46.799+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9753 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:46.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:46.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:46.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:46.921+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:46.876+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:46.876+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:46.876+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:46.876+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:47.828 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:47.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:47.828 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:47.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:47.829 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:47.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:47.829 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:47.463+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:47.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:47.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:47.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:47.946+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:47.828+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:47.828+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:47.828+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:47.828+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:48.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:48.422+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:48.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:48.422+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:48.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:48.422+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:48.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:48.422+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:48.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:48.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:48.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:48.980+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:48.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:48.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:48.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:48.863+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:49.832 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:49.403+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:49.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:49.403+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:49.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:49.403+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:49.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:49.403+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:49.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:49.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:49.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:49.984+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:49.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:49.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:49.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:49.832+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:50.829 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:50.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:50.829 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:50.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:50.829 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:50.379+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:50.829 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:50.379+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:50.828+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:50.828+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:50.828+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:50.828+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:51.028+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:51.028+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:51.028+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:51.028+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:51.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:51.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:51.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:51.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:51.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:51.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:51.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:51.413+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:51.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:51.800+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9847 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:51.930 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:51.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:51.930 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:51.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:51.930 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:51.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:51.930 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:51.837+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:52.057+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:52.057+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:52.057+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:52.057+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:52.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:52.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:52.418+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:52.418+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:52.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:52.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:52.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:52.878+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:53.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:53.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:53.088+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:53.088+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:53.838 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:53.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:53.838 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:53.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:53.838 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:53.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:53.838 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:53.425+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:54.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:53.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:54.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:53.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:54.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:53.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:54.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:53.837+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:54.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:54.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:54.110+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:54.110+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:54.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:54.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:54.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:54.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:54.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:54.453+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:54.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:54.453+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:55.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:54.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:55.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:54.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:55.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:54.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:55.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:54.856+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:55.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:55.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:55.109+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:55.109+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:55.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:55.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:55.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:55.489+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:56.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:56.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:56.111+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:56.111+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:55.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:55.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:55.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:55.875+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:56.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:56.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:56.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:56.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:56.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:56.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:56.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:56.440+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:57.074 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:55:56.802+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 9942 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:55:57.075 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:56.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:57.075 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:56.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:57.075 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:56.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:57.075 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:56.862+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:57.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:57.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:57.074+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:57.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:57.074+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:57.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:57.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:57.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:57.440+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:58.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:58.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:58.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:58.117+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:57.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:57.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:57.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:57.877+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:58.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:58.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:58.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:58.412+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:59.096+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:59.096+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:59.096+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:55:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:55:59.096+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:58.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:58.875+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:58.875+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:55:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:58.875+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:55:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:59.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:59.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:59.374+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:55:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:55:59.375+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:00.065+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:00.065+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:00.065+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:00.065+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:59.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:59.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:59.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:55:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:55:59.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:00.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:00.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:00.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:00.412+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:00.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:00.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:00.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:00.893+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:01.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:01.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:01.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:01.033+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:01.805 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:01.459+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:01.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:01.459+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:01.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:01.459+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:01.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:01.459+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:02.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:01.804+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10038 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:02.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:02.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:02.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:02.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:02.044+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:02.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:01.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:02.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:01.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:02.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:01.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:02.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:01.914+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:02.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:02.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:02.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:02.428+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:03.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:03.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:03.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:03.045+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:02.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:02.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:02.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:02.959+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:03.382+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:03.382+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:03.382+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:03.382+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:04.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:03.911+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:04.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:03.911+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:04.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:03.911+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:04.336 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:03.911+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:04.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:04.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:04.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:04.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.032+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:04.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:04.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:04.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:04.333+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:05.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:05.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:05.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:05.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:04.989+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:05.309 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:04.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:05.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:04.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:05.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:04.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:05.310 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:04.922+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:05.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:05.307+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:05.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:05.307+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:05.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:05.307+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:05.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:05.307+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:06.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:05.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:06.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:05.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:06.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:05.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:06.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:05.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:06.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.024+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:06.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.024+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:06.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.024+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:06.336 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.024+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:06.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:06.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:06.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:06.334+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:07.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:06.805+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10134 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:06.977+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:06.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:06.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:06.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:06.880+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:07.372+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:07.372+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:07.372+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:07.372+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:08.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:07.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:08.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:07.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:08.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:07.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:08.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:07.880+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:08.333 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:07.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:08.333 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:07.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:08.333 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:07.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:08.333 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:07.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:08.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:08.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:08.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:08.331+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:08.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:08.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:08.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:08.973+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:08.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:08.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:08.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:08.854+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:09.293+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:09.293+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:09.293+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:09.293+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:10.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:09.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:10.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:09.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:10.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:09.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:10.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:09.900+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:10.261 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:09.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:10.261 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:09.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:10.261 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:09.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:10.261 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:09.932+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:10.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:10.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:10.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:10.259+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:11.272 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:10.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:11.272 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:10.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:11.273 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:10.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:11.273 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:10.975+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:11.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:10.886+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:11.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:10.886+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:11.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:10.886+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:11.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:10.886+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:11.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:11.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:11.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:11.271+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:11.934 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:11.808+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10229 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:11.935 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:11.891+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:11.935 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:11.891+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:11.935 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:11.891+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:11.935 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:11.892+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:12.223 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:11.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:12.224 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:11.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:12.224 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:11.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:12.224 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:11.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:12.223+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:12.223+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:12.223+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:12.223+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:13.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:12.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:13.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:12.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:13.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:12.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:13.212 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:12.898+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:13.212 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:12.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:13.212 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:12.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:13.212 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:12.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:13.212 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:12.983+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:13.567 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:13.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:13.567 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:13.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:13.567 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:13.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:13.567 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:13.210+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:14.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:13.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:14.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:13.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:14.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:13.996+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:14.251 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:13.996+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:14.252 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:13.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:14.252 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:13.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:14.252 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:13.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:14.252 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:13.896+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:14.250+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:14.250+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:14.250+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:14.250+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:14.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:14.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:14.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:14.999+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:14.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:14.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:14.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:14.856+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:15.258+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:15.258+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:15.258+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:15.258+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:16.287 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:16.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:16.287 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:16.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:16.287 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:16.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:16.288 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:16.045+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:16.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:15.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:16.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:15.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:16.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:15.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:16.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:15.882+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:16.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:16.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:16.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:16.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:16.809+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10324 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:17.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:17.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:17.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:17.021+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:17.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:16.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:17.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:16.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:17.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:16.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:17.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:16.844+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:17.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:17.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:17.335+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:17.335+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.021+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.021+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:17.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:17.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:17.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:17.857+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:18.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:18.339+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:18.339+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:18.339+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:18.339+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:19.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:18.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:19.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:18.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:19.362 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:18.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:19.362 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:18.896+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:19.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:19.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:19.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:19.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:18.984+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:19.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:19.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:19.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:19.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:20.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:19.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:20.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:19.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:20.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:19.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:20.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:19.894+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:20.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:19.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:20.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:19.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:20.361 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:19.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:20.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:19.980+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:20.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:20.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:20.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:20.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:20.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:20.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:20.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:20.944+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:21.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:21.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:21.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:21.015+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:21.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:21.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:21.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:21.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:21.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:21.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:21.675 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:21.394+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:21.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:21.811+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10420 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:21.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:21.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:21.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:21.975+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:22.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:22.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:22.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:22.043+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:22.043+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:22.376+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:22.376+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:22.376+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:22.377+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:23.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:23.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:23.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.023+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:23.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.023+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:23.330 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:23.330 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:23.330 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:23.330 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.027+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:23.359+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:23.359+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:23.359+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:23.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:23.974+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:23.989+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:24.406+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:24.406+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:24.406+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:24.406+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:24.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:24.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:24.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:24.961+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:25.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:25.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:25.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:25.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:25.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:25.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:25.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:25.426+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:26.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:26.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:26.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:26.041+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:25.949+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:25.949+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:25.949+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:25.950+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:26.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:26.419+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:26.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:26.419+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:26.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:26.419+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:26.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:26.419+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:26.813+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10516 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:26.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:26.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:26.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:26.942+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:27.030+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:27.030+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:27.030+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:27.030+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:27.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:27.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:27.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:27.404+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:27.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:27.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:27.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:27.927+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:28.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:28.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:28.076+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:28.076+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:28.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:28.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:28.405+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:28.405+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:28.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:28.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:28.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:28.920+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:29.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:29.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:29.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:29.123+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:29.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:29.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:29.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:29.392+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:30.173 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:29.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:30.174 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:29.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:30.174 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:29.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:30.174 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:29.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:30.433 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:30.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:30.433 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:30.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:30.433 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:30.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:30.433 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:30.173+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:30.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:30.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:30.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:30.432+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:31.193 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:30.923+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:31.193 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:30.923+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:31.194 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:30.923+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:31.194 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:30.923+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:31.471 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:31.192+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:31.472 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:31.192+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:31.472 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:31.192+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:31.472 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:31.193+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:31.816 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:31.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:31.816 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:31.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:31.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:31.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:31.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:31.471+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:31.815+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10611 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:31.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:31.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:31.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:32.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:31.885+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:32.495 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:32.165+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:32.495 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:32.165+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:32.496 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:32.165+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:32.496 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:32.166+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:32.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:32.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:32.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:32.494+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:33.154 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:32.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:33.155 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:32.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:33.155 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:32.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:33.155 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:32.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:33.481 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:33.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:33.481 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:33.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:33.482 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:33.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:33.482 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:33.154+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:33.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:33.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:33.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:33.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:33.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:33.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:33.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:33.481+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:34.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:33.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:33.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:33.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:33.864+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:34.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:34.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:34.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:34.435+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:34.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:34.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:34.203+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:34.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:34.204+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:35.197 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:34.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:35.198 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:34.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:35.198 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:34.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:35.198 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:34.873+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:35.617 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:35.197+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:35.617 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:35.197+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:35.617 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:35.197+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:35.617 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:35.197+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:35.617 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:35.419+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:35.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:35.419+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:35.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:35.419+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:35.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:35.419+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:36.179 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:35.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:36.179 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:35.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:36.180 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:35.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:36.180 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:35.887+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:36.465 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:36.178+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:36.466 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:36.178+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:36.466 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:36.178+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:36.466 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:36.178+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:36.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:36.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:36.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:36.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:36.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:36.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:36.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:36.465+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:36.817+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10706 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:36.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:36.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:36.857+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:36.858+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:37.487 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:37.195+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:37.488 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:37.195+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:37.488 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:37.195+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:37.488 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:37.195+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:37.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:37.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:37.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:37.487+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:38.237 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:37.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:38.238 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:37.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:38.238 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:37.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:38.238 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:37.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:38.502 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:38.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:38.502 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:38.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:38.502 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:38.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:38.502 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:38.237+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:38.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:38.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:38.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:38.502+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:39.206 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:38.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:39.206 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:38.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:39.206 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:38.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:39.206 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:38.908+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:39.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:39.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:39.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:39.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:39.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:39.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:39.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:39.206+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:39.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:39.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:39.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:39.530+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:40.194 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:39.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:40.194 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:39.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:40.194 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:39.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:40.194 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:39.916+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:40.529 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:40.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:40.529 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:40.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:40.529 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:40.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:40.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:40.194+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:40.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:40.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:40.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:40.529+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:41.165 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:40.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:41.165 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:40.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:41.165 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:40.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:41.165 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:40.892+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:41.525 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:41.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:41.525 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:41.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:41.526 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:41.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:41.526 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:41.164+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:41.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:41.524+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:41.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:41.524+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:41.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:41.524+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:41.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:41.525+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:41.819+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10802 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:41.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:41.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:41.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:41.895+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:42.540 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:42.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:42.540 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:42.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:42.540 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:42.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:42.540 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:42.148+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:42.855 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:42.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:42.855 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:42.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:42.855 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:42.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:42.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:42.538+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:42.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:42.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:42.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:42.853+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:43.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:43.182+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:43.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:43.182+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:43.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:43.182+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:43.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:43.182+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:43.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:43.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:43.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:43.561+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:44.207 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:43.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:44.208 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:43.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:44.208 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:43.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:44.208 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:43.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:44.551 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:44.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:44.551 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:44.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:44.552 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:44.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:44.552 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:44.206+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:44.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:44.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:44.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:44.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:44.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:44.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:44.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:44.549+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:44.909+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:44.909+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:44.909+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:44.909+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:45.506 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:45.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:45.506 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:45.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:45.506 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:45.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:45.507 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:45.194+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:45.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:45.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:45.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:45.505+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:46.224 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:45.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:46.224 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:45.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:46.224 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:45.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:46.224 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:45.904+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:46.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:46.223+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:46.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:46.223+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:46.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:46.223+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:46.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:46.223+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:46.821 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:46.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:46.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:46.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:46.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:46.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:46.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:46.553+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:47.120 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:46.819+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10898 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:46.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:46.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:46.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:46.913+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:47.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:47.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:47.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:47.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:47.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:47.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:47.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:47.194+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:47.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:47.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:47.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:47.601+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:48.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:47.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:48.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:47.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:48.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:47.918+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:48.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:47.918+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:48.553 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:48.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:48.554 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:48.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:48.554 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:48.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:48.554 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:48.244+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:48.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:48.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:48.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:48.553+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:49.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:48.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:49.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:48.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:49.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:48.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:49.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:48.921+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:49.560 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:49.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:49.561 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:49.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:49.561 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:49.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:49.561 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:49.265+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:49.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:49.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:49.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:49.560+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:50.247 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:49.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:50.247 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:49.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:50.247 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:49.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:50.247 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:49.951+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:50.571 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:50.246+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:50.572 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:50.246+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:50.572 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:50.246+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:50.572 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:50.246+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:50.571+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:50.571+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:50.571+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:50.571+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:51.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:50.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:51.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:50.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:51.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:50.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:51.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:50.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:51.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:51.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:51.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:51.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:51.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:51.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:51.569 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:51.264+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:51.823 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:51.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:51.823 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:51.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:51.823 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:51.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:51.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:51.568+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:52.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:51.822+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 10993 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:52.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:51.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:52.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:51.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:51.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:51.959+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:52.543 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:52.268+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:52.543 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:52.268+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:52.544 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:52.268+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:52.544 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:52.268+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:52.542+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:52.542+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:52.542+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:52.542+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:53.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:52.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:53.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:52.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:53.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:52.975+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:53.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:52.975+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:53.557 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:53.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:53.558 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:53.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:53.558 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:53.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:53.558 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:53.278+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:53.557+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:53.557+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:53.557+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:53.557+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:54.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:53.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:54.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:53.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:54.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:53.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:54.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:53.931+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:54.529 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:54.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:54.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:54.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:54.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:54.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:54.530 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:54.260+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:54.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:54.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:54.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:54.529+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:55.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:54.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:55.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:54.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:55.267 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:54.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:55.267 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:54.940+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:55.536 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:55.265+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:55.536 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:55.265+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:55.536 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:55.265+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:55.536 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:55.266+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:55.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:55.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:55.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:55.535+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:56.250 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:55.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:56.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:55.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:56.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:55.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:56.251 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:55.986+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:56.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:56.250+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:56.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:56.250+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:56.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:56.250+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:56.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:56.250+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:56.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:56.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:56.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:56.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:56.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:56.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:56.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:56.563+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:57.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:56:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:56:56.824+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11088 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:56:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.012+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:57.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:57.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:57.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:57.515+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:57.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:57.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:57.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:57.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:57.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:57.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:57.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:57.288+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:58.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:58.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:58.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.974+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:58.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:57.974+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-10T06:56:58.268 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-10T06:56:58.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:56:58.268 DEBUG:teuthology.orchestra.run.smithi073:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-10T06:56:58.336 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/9-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.337 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/10-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.337 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.337 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.338 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.338 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.338 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.338 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.339 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/12-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.339 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.339 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/13-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.339 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/4-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.340 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/11-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.340 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive/11-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.340 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_receive: Cannot utime: Input/output error 2024-02-10T06:56:58.340 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init: Cannot mkdir: Input/output error 2024-02-10T06:56:58.341 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.341 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.341 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.341 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_init/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.342 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval: Cannot mkdir: Input/output error 2024-02-10T06:56:58.342 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.342 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.342 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.343 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.343 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.343 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/4-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.343 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/speculative: Cannot mkdir: Input/output error 2024-02-10T06:56:58.344 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_rr_get_interval/speculative/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.344 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared: Cannot mkdir: Input/output error 2024-02-10T06:56:58.344 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.344 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/5-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.345 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.345 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.345 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.345 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_getpshared/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.345 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending: Cannot mkdir: Input/output error 2024-02-10T06:56:58.346 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.346 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/1-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.346 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.346 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.347 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.347 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigpending/1-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.347 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstackaddr: Cannot mkdir: Input/output error 2024-02-10T06:56:58.347 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstackaddr/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.348 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstackaddr/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.348 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit: Cannot mkdir: Input/output error 2024-02-10T06:56:58.348 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.349 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/6-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.349 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.349 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/1-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.350 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/testfrmw.c: Cannot open: Input/output error 2024-02-10T06:56:58.350 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/testfrmw.h: Cannot open: Input/output error 2024-02-10T06:56:58.350 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.350 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.350 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.351 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.351 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.351 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/2-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.351 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.352 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/threads_scenarii.c: Cannot open: Input/output error 2024-02-10T06:56:58.352 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_exit/3-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.352 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock: Cannot mkdir: Input/output error 2024-02-10T06:56:58.352 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.353 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.353 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/testfrmw.c: Cannot open: Input/output error 2024-02-10T06:56:58.353 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/testfrmw.h: Cannot open: Input/output error 2024-02-10T06:56:58.354 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.354 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.355 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.355 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/5-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.355 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_unlock/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.355 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy: Cannot mkdir: Input/output error 2024-02-10T06:56:58.356 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.356 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.356 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.356 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.357 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlockattr_destroy/3-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.357 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify: Cannot mkdir: Input/output error 2024-02-10T06:56:58.357 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.358 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/9-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.358 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.358 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/6-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.359 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.359 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.359 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.359 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.359 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/7-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.359 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_notify/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield: Cannot mkdir: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.360 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_yield/3-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync: Cannot mkdir: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/9-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/4-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/10-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.361 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-4.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/14-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.362 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/12-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/13-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/8-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_fsync/11-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack: Cannot mkdir: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.363 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/5-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/3-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.364 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstack/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init: Cannot mkdir: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_init/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime: Cannot mkdir: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.365 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mktime/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send: Cannot mkdir: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/9-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/4-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/10-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.366 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/14-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/5-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.367 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/12-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/4-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/13-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/11-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.368 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/11-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.369 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_send/3-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.369 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return: Cannot mkdir: Input/output error 2024-02-10T06:56:58.369 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.369 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.369 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.369 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.370 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.370 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.370 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_return/3-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.370 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset: Cannot mkdir: Input/output error 2024-02-10T06:56:58.371 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.371 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/9-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.371 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/10-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.371 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.372 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.372 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.372 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.373 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.373 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.373 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.373 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.374 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigset/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.374 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init: Cannot mkdir: Input/output error 2024-02-10T06:56:58.374 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.375 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.375 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.375 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.375 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.376 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_init/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.376 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock: Cannot mkdir: Input/output error 2024-02-10T06:56:58.376 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.377 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/4-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.377 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.377 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/1-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.377 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/testfrmw.c: Cannot open: Input/output error 2024-02-10T06:56:58.378 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/testfrmw.h: Cannot open: Input/output error 2024-02-10T06:56:58.378 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.378 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.378 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.379 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/4-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.379 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_trylock/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.379 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr: Cannot mkdir: Input/output error 2024-02-10T06:56:58.379 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.380 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.380 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/5-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.380 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/6-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.380 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.381 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/1-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.381 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.381 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/2-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.381 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.382 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/speculative: Cannot mkdir: Input/output error 2024-02-10T06:56:58.382 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_getattr/speculative/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.382 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join: Cannot mkdir: Input/output error 2024-02-10T06:56:58.382 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.383 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/6-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.383 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.383 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/1-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.383 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/testfrmw.c: Cannot open: Input/output error 2024-02-10T06:56:58.383 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/testfrmw.h: Cannot open: Input/output error 2024-02-10T06:56:58.384 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.384 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.384 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.384 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.385 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.385 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/threads_scenarii.c: Cannot open: Input/output error 2024-02-10T06:56:58.385 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/6-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.385 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/speculative: Cannot mkdir: Input/output error 2024-02-10T06:56:58.386 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_join/speculative/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.386 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize: Cannot mkdir: Input/output error 2024-02-10T06:56:58.386 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.386 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.387 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.387 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstacksize/2-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.387 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched: Cannot mkdir: Input/output error 2024-02-10T06:56:58.387 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.387 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.388 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.388 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getinheritsched/2-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.388 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait: Cannot mkdir: Input/output error 2024-02-10T06:56:58.389 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.389 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/6-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.389 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.390 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/testfrmw.c: Cannot open: Input/output error 2024-02-10T06:56:58.390 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/testfrmw.h: Cannot open: Input/output error 2024-02-10T06:56:58.390 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.390 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.390 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwait/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime: Cannot mkdir: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.391 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/strftime/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max: Cannot mkdir: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.392 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/3-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_max/1-4.c: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete: Cannot mkdir: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.393 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/1-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/3-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/4-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/2-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/speculative: Cannot mkdir: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/speculative/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.394 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_delete/speculative/5-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime: Cannot mkdir: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/gmtime/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy: Cannot mkdir: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.395 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_destroy/4-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock: Cannot mkdir: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.396 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/3-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_tryrdlock/2-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam: Cannot mkdir: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/27-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/3-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/9-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/20-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/10-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.397 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/19-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/coverage.txt: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/18-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/25-4.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/25-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/14-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.398 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/2-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.399 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/5-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.399 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/6-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.399 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/1-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.399 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-02-10T06:56:58.399 INFO:tasks.workunit.client.0.smithi073.stderr:gzip: tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.399 INFO:tasks.workunit.client.0.smithi073.stderr:stdin: Stale file handle 2024-02-10T06:56:58.400 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/15-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.400 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/8-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.400 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/24-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.401 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi073.front.sepia.ceph.com 2024-02-10T06:56:58.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:56:58.401 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-10T06:56:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-10T06:56:58.403 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/assertions.xml: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/12-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/7-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/21-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-6.c: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-7.c: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/2-2.c: Cannot open: Input/output error 2024-02-10T06:56:58.404 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/25-3.c: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/13-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-4.c: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/4-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/22-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/26-1.c: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/11-1.sh: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_setparam/23-5.c: Cannot open: Input/output error 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: Unexpected EOF in archive 2024-02-10T06:56:58.405 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces: Cannot utime: Input/output error 2024-02-10T06:56:58.406 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance: Cannot utime: Input/output error 2024-02-10T06:56:58.406 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite: Cannot utime: Input/output error 2024-02-10T06:56:58.406 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-02-10T06:56:58.406 INFO:tasks.workunit.client.0.smithi073.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-02-10T06:56:58.406 INFO:tasks.workunit.client.0.smithi073.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:56:58.406 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-10T06:56:58.407 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-10T06:56:58.470 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-10T06:56:58.470 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:56:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:58.508+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:58.508+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:58.508+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:58.508+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:58.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:58.289+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:58.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:58.289+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:58.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:58.289+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:58.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:58.289+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:59.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:58.956+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:59.276 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:58.956+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:59.276 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:58.956+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:56:59.276 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:58.956+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:59.286+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:59.286+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:59.286+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:56:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:56:59.286+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:56:59.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:59.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:59.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:59.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:59.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:59.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:56:59.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:56:59.472+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:00.331 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:59.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:00.331 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:59.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:00.331 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:59.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:00.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:56:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:56:59.964+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:00.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:00.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:00.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:00.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:00.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:00.330+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:00.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:00.330+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:00.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:00.330+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:00.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:00.330+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:01.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:00.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:01.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:00.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:01.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:00.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:01.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:00.951+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:01.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:01.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:01.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:01.505+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:01.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:01.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:01.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:01.331+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:01.940 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:01.826+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11184 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:02.368 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:01.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:02.369 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:01.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:02.369 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:01.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:02.369 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:01.940+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:02.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:02.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:02.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:02.523+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:02.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:02.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:02.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:02.368+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:03.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:02.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:03.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:02.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:03.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:02.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:03.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:02.899+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:03.572 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:03.490+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:03.572 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:03.490+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:03.572 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:03.490+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:03.573 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:03.490+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:03.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:03.352+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:03.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:03.352+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:03.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:03.352+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:03.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:03.352+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:04.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:03.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:04.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:03.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:04.357 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:03.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:04.357 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:03.887+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:04.489+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:04.489+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:04.489+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:04.489+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:04.355+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:04.355+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:04.355+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:04.355+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:04.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:04.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:04.856+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:04.856+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:05.818 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:05.389+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:05.818 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:05.389+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:05.818 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:05.389+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:05.819 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:05.389+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:05.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:05.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:05.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:05.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:05.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:05.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:05.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:05.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:05.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:05.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:05.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:05.817+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:06.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:06.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:06.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:06.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:06.437+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:06.437+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:06.437+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:06.437+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:07.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:06.828+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11280 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:06.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:06.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:06.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:06.788+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:07.783 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:07.471+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:07.783 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:07.472+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:07.783 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:07.472+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:07.783 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:07.472+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:07.784 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:07.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:07.784 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:07.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:07.784 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:07.560+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:07.784 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:07.560+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:07.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:07.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:07.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:07.782+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:08.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:08.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:08.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:08.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:08.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:08.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:08.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:08.594+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:08.793 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:08.450+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:08.793 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:08.450+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:08.793 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:08.450+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:08.794 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:08.450+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:08.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:08.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:08.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:08.791+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:09.797 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:09.464+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:09.798 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:09.464+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:09.798 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:09.464+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:09.798 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:09.464+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:09.798 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:09.546+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:09.798 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:09.546+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:09.799 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:09.546+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:09.799 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:09.546+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:09.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:09.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:09.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:09.797+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:10.757 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:10.452+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:10.757 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:10.452+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:10.757 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:10.452+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:10.757 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:10.452+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:10.758 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:10.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:10.758 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:10.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:10.758 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:10.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:10.758 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:10.513+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:10.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:10.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:10.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:10.756+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:11.679 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:11.504+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:11.679 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:11.504+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:11.679 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:11.504+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:11.679 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:11.504+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:11.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:11.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:11.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:11.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:11.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:11.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:11.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:11.404+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:11.956 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:11.830+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11375 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:11.956 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:11.785+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:11.956 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:11.785+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:11.956 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:11.785+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:11.957 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:11.785+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:12.812 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:12.434+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:12.813 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:12.434+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:12.813 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:12.434+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:12.813 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:12.434+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:12.813 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:12.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:12.813 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:12.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:12.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:12.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:12.814 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:12.525+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:12.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:12.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:12.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:12.812+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:13.833 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:13.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:13.834 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:13.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:13.834 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:13.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:13.834 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:13.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:13.834 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:13.422+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:13.834 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:13.422+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:13.835 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:13.422+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:13.835 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:13.422+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:13.833+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:13.833+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:13.833+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:13.833+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:14.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:14.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:14.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:14.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:14.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:14.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:14.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:14.463+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:14.819 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:14.417+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:14.820 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:14.417+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:14.820 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:14.417+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:14.820 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:14.417+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:14.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:14.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:14.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:14.818+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:15.799 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:15.410+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:15.799 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:15.410+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:15.799 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:15.410+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:15.799 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:15.410+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:15.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:15.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:15.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:15.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:15.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:15.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:15.800 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:15.488+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:16.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:15.798+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:15.798+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:15.798+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:15.798+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:16.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:16.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:16.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:16.477+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:16.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:16.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:16.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:16.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:16.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:16.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:16.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:16.368+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:16.832+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11470 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:16.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:16.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:16.802+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:17.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:16.802+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:17.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:17.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:17.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:17.495+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:17.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:17.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:17.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:17.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:17.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:17.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:17.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:17.340+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:18.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:17.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:17.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:17.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:17.762+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:18.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:18.296+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:18.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:18.296+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:18.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:18.296+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:18.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:18.296+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:18.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:18.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:18.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:18.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:18.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:18.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:18.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:18.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:18.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:18.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:18.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:18.724+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:19.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:19.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:19.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:19.541+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:19.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:19.334+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:19.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:19.334+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:19.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:19.334+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:19.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:19.334+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:19.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:19.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:19.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:19.718+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:20.516 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:20.370+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:20.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:20.370+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:20.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:20.370+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:20.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:20.370+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:20.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:20.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:20.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:20.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:20.516+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:20.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:20.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:20.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:20.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:20.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:20.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:20.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:20.752+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:21.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:21.410+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:21.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:21.410+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:21.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:21.410+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:21.680 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:21.410+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:21.680 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:21.534+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:21.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:21.534+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:21.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:21.534+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:21.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:21.534+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:21.959 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:21.833+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11566 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:21.959 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:21.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:21.960 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:21.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:21.960 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:21.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:21.960 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:21.762+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:22.363+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:22.363+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:22.363+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:22.363+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:22.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:22.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:22.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:22.523+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:22.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:22.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:22.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:22.727+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:23.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:23.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:23.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:23.512+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:23.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:23.351+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:23.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:23.351+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:23.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:23.351+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:23.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:23.351+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:23.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:23.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:23.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:23.746+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:24.718 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:24.391+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:24.718 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:24.391+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:24.719 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:24.391+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:24.719 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:24.391+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:24.719 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:24.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:24.719 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:24.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:24.719 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:24.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:24.719 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:24.553+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:24.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:24.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:24.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:24.717+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:25.699 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:25.394+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:25.699 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:25.394+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:25.699 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:25.394+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:25.699 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:25.394+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:25.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:25.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:25.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:25.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:25.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:25.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:25.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:25.551+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:26.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:25.697+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:25.697+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:25.697+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:25.697+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:26.702 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:26.402+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:26.702 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:26.402+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:26.702 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:26.402+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:26.702 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:26.402+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:26.702 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:26.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:26.702 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:26.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:26.703 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:26.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:26.703 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:26.509+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:26.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:26.834+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11662 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:26.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:26.700+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:26.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:26.700+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:26.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:26.700+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:26.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:26.700+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:27.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:27.417+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:27.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:27.417+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:27.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:27.417+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:27.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:27.417+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:27.751 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:27.464+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:27.752 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:27.464+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:27.752 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:27.464+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:27.752 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:27.464+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:27.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:27.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:27.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:27.750+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:28.747 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:28.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:28.747 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:28.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:28.748 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:28.420+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:28.748 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:28.420+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:28.748 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:28.467+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:28.748 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:28.467+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:28.748 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:28.467+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:28.748 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:28.467+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:29.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:28.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:28.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:28.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:28.747+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:29.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:29.381+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:29.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:29.381+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:29.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:29.381+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:29.745 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:29.381+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:29.745 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:29.431+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:29.746 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:29.431+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:29.746 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:29.431+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:29.746 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:29.431+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:29.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:29.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:29.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:29.744+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:30.476 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:30.384+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:30.476 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:30.384+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:30.476 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:30.384+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:30.476 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:30.385+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:30.731 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:30.475+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:30.731 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:30.475+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:30.732 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:30.475+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:30.732 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:30.475+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:30.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:30.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:30.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:30.731+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:31.680 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:31.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:31.680 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:31.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:31.680 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:31.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:31.680 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:31.428+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:31.681 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:31.524+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:31.681 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:31.524+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:31.681 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:31.524+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:31.681 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:31.525+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:31.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:31.837+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11757 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:31.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:31.696+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:31.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:31.696+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:31.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:31.696+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:31.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:31.696+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:32.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:32.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:32.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:32.457+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:32.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:32.682+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:32.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:32.682+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:32.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:32.682+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:32.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:32.682+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:32.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:32.487+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:32.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:32.487+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:32.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:32.487+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:32.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:32.487+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:33.703 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:33.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:33.704 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:33.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:33.704 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:33.423+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:33.704 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:33.423+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:33.704 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:33.490+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:33.704 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:33.490+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:33.705 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:33.490+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:33.705 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:33.490+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:33.984 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:33.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:33.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:33.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:33.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:33.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:33.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:33.702+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:34.497+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:34.497+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:34.497+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:34.497+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:34.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:34.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:34.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:34.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:34.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:34.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:34.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:34.434+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:34.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:34.659+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:34.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:34.659+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:34.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:34.659+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:34.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:34.659+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:35.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:35.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:35.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:35.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:35.484+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:35.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:35.704+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:35.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:35.704+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:35.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:35.704+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:35.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:35.705+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:35.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:35.542+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:35.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:35.542+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:35.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:35.542+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:35.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:35.543+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:36.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:36.518+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:36.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:36.518+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:36.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:36.518+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:36.841 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:36.518+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:36.841 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:36.680+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:36.841 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:36.680+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:36.841 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:36.680+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:36.841 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:36.680+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:36.842 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:36.557+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:36.842 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:36.557+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:36.842 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:36.557+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:36.842 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:36.557+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:36.839+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11852 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:37.657 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:37.556+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:37.657 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:37.556+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:37.658 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:37.556+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:37.658 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:37.556+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:37.659 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:37.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:37.659 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:37.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:37.659 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:37.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:37.659 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:37.527+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:37.656+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:37.656+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:37.656+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:37.656+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:38.565+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:38.565+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:38.565+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:38.566+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:38.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:38.667+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:38.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:38.667+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:38.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:38.667+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:38.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:38.667+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:38.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:38.598+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:38.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:38.598+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:38.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:38.598+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:38.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:38.598+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:39.534+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:39.534+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:39.534+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:39.534+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:39.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:39.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:39.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:39.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:39.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:39.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:39.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:39.671+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:39.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:39.626+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:39.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:39.626+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:39.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:39.626+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:39.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:39.626+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:40.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:40.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:40.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:40.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:40.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:40.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:40.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:40.677+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:40.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:40.677+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:40.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:40.677+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:40.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:40.677+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:40.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:40.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:40.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:40.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:40.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:40.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:40.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:40.644+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:41.842 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:41.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:41.842 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:41.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:41.842 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:41.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:41.842 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:41.474+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:41.843 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:41.638+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:41.843 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:41.638+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:41.843 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:41.638+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:41.843 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:41.638+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:41.843 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:41.620+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:41.844 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:41.620+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:41.844 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:41.620+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:41.844 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:41.620+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:42.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:41.841+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 11948 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:42.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:42.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:42.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:42.471+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:42.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:42.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:42.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:42.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:42.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:42.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:42.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:42.645+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:42.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:42.602+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:42.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:42.602+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:42.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:42.602+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:42.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:42.602+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:43.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:43.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:43.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:43.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:43.671+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:43.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:43.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:43.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:43.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:43.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:43.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:43.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:43.647+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:43.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:43.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:43.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:43.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:43.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:43.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:43.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:43.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:44.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:44.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:44.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:44.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:44.484+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:44.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:44.629+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:44.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:44.629+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:44.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:44.629+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:44.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:44.629+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:44.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:44.673+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:44.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:44.673+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:44.874 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:44.673+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:44.874 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:44.673+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:45.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:45.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:45.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:45.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:45.503+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:45.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:45.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:45.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:45.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:45.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:45.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:45.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:45.711+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:45.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:45.628+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:45.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:45.628+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:45.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:45.628+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:45.874 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:45.628+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:46.844 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:46.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:46.844 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:46.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:46.844 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:46.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:46.845 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:46.613+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:46.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:46.668+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:46.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:46.668+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:46.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:46.668+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:46.845 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:46.668+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:46.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:46.508+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:46.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:46.508+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:46.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:46.508+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:46.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:46.508+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:46.843+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12044 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:47.710 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:47.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:47.710 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:47.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:47.711 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:47.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:47.711 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:47.551+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:47.711 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:47.579+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:47.712 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:47.579+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:47.712 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:47.579+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:47.712 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:47.579+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:47.709+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:47.709+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:47.709+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:47.709+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:48.537+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:48.537+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:48.537+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:48.537+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:48.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:48.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:48.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:48.507+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:48.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:48.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:48.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:48.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:48.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:48.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:48.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:48.685+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:49.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:49.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:49.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:49.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:49.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:49.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:49.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:49.564+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:49.552+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:49.552+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:49.552+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:49.552+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:49.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:49.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:49.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:49.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:49.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:49.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:49.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:49.725+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:50.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:50.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:50.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:50.581+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:50.741+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:50.741+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:50.741+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:50.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:50.741+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:50.595+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:50.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:50.595+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:50.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:50.595+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:50.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:50.595+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:51.846 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:51.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:51.846 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:51.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:51.846 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:51.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:51.847 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:51.632+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:51.847 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:51.580+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:51.847 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:51.580+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:51.847 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:51.580+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:51.847 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:51.580+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:51.848 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:51.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:51.848 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:51.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:51.848 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:51.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:51.848 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:51.724+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:52.120 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:51.845+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12139 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:52.729 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:52.627+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:52.729 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:52.627+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:52.729 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:52.627+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:52.730 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:52.627+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:52.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:52.614+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:52.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:52.614+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:52.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:52.614+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:52.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:52.614+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:52.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:52.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:52.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:52.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:53.571+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:53.571+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:53.571+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:53.571+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:53.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:53.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:53.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:53.686+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:53.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:53.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:53.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:53.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:53.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:53.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:53.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:53.632+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:54.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:54.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:54.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:54.598+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:54.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:54.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:54.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:54.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:54.677+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:54.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:54.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:54.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:54.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:54.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:54.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:54.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:54.640+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:55.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:55.602+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:55.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:55.602+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:55.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:55.602+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:55.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:55.602+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:55.629+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:55.629+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:55.629+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:55.629+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:55.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:55.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:55.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:55.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:55.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:55.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:55.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:55.619+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:56.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:57:56.847+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12234 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:57:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:56.619+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:56.619+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:56.619+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:56.619+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:56.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:56.633+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:56.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:56.633+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:56.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:56.633+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:56.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:56.633+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:56.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:56.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:56.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:56.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:56.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:56.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:56.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:56.632+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:57.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:57.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:57.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:57.644+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:57.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:57.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:57.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:57.637+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:57.649+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:57.649+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:57.649+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:58.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:57.649+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:58.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:58.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:58.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:57:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:58.685+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:58.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:58.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:58.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:58.690+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:58.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:58.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:58.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:57:59.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:58.668+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:59.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:59.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:59.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:57:59.729+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:59.635+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:59.635+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:59.635+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:57:59.635+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:59.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:59.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:59.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:00.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:57:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:57:59.722+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:00.679+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:00.679+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:00.679+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:00.679+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:00.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:00.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:00.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:00.746+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:00.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:00.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:00.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:01.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:00.772+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:01.975 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:01.848+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12330 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:01.975 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:01.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:01.975 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:01.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:01.975 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:01.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:01.976 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:01.640+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:01.976 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:01.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:01.976 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:01.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:01.976 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:01.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:01.976 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:01.724+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:01.977 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:01.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:01.977 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:01.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:01.977 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:01.772+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:01.977 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:01.773+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:02.595+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:02.595+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:02.595+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:02.595+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:02.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:02.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:02.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:02.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:02.756+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:02.875 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:02.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:02.875 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:02.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:02.875 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:02.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:02.876 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:02.764+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:03.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:03.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:03.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:03.637+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:03.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:03.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:03.762+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:03.762+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:03.716+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:04.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:03.716+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:04.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:03.716+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:04.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:03.716+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:04.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:04.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:04.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:04.653+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:04.732+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:04.732+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:04.732+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:04.732+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:04.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:04.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:05.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:04.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:05.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:04.739+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:05.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:05.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:05.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:05.700+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:05.766+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:05.766+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:05.766+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:05.766+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:05.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:06.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:05.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:06.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:05.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:06.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:05.771+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:06.975 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:06.850+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12426 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:06.976 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:06.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:06.976 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:06.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:06.976 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:06.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:06.976 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:06.706+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:06.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:06.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:06.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:06.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:06.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:06.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:06.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:06.763+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:06.977 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:06.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:06.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:06.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:06.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:06.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:06.978 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:06.802+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:07.657+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:07.657+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:07.657+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:07.657+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:07.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:07.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:07.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:07.812+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:07.830+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:07.830+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:08.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:07.830+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:08.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:07.831+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:08.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:08.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:08.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:08.613+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:08.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:08.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:08.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:08.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:08.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:08.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:08.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:08.769+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:08.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:08.852+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:08.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:08.852+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:08.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:08.852+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:08.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:08.852+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:09.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:09.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:09.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:09.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:09.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:09.605+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:09.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:09.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:09.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:09.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:09.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:09.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:09.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:09.778+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:09.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:09.835+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:09.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:09.835+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:09.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:09.835+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:09.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:09.836+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:10.850 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:10.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:10.850 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:10.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:10.850 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:10.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:10.850 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:10.562+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:10.850 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:10.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:10.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:10.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:10.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:10.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:10.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:10.751+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:10.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:10.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:10.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:10.869+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:11.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:11.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:11.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:11.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:11.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:11.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:11.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:11.514+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:12.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:11.853+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12521 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:11.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:11.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:11.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:11.782+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:11.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:12.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:11.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:12.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:11.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:12.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:11.854+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:12.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:12.492+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:12.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:12.492+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:12.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:12.492+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:12.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:12.492+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:12.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:12.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:12.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:12.803+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:13.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:12.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:13.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:12.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:13.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:12.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:13.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:12.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:13.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:13.490+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:13.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:13.490+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:13.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:13.490+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:13.818 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:13.490+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:14.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:13.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:14.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:13.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:14.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:13.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:14.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:13.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:13.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:13.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:13.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:13.817+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:14.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:14.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:14.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:14.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:14.782 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:14.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:14.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:14.455+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:14.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:14.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:14.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:14.919+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:14.781+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:14.781+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:14.781+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:14.781+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:15.754 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:15.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:15.755 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:15.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:15.755 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:15.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:15.755 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:15.495+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:15.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:15.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:15.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:15.937+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:15.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:15.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:15.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:15.754+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:16.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:16.489+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:16.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:16.489+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:16.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:16.489+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:16.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:16.490+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:16.857 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:16.736+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:16.857 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:16.736+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:16.858 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:16.736+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:16.858 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:16.736+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:17.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:16.855+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12616 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:16.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:16.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:16.893+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:16.893+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:17.765 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:17.519+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:17.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:17.519+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:17.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:17.519+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:17.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:17.519+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:17.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:17.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:17.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:17.870+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:18.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:17.764+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:17.764+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:17.764+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:17.765+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:18.865 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:18.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:18.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:18.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:18.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:18.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:18.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:18.543+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:18.866 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:18.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:18.866 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:18.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:18.867 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:18.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:18.867 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:18.775+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:18.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:18.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:18.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:18.865+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:19.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:19.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:19.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:19.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:19.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:19.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:19.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:19.502+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:19.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:19.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:19.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:19.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:19.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:19.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:19.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:19.755+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:20.711 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:20.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:20.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:20.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:20.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:20.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:20.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:20.457+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:20.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:20.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:20.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:20.711+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:20.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:20.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:20.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:20.799+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:21.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:21.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:21.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:21.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:21.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:21.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:21.846 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:21.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:21.846 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:21.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:21.846 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:21.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:21.846 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:21.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:21.846 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:21.712+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:21.857+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12712 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:22.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:21.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:22.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:21.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:22.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:21.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:21.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:22.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:22.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:22.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:22.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:22.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:22.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:22.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:22.514+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:22.826 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:22.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:22.826 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:22.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:22.826 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:22.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:22.827 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:22.746+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:22.825+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:22.825+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:22.825+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:22.825+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:23.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:23.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:23.803 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:23.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:23.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:23.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:23.804 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:23.517+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:23.804 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:23.737+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:23.804 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:23.737+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:23.804 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:23.737+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:23.804 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:23.737+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:23.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:23.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:23.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:23.803+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:24.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:24.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:24.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:24.563+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:24.796+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:24.796+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:24.796+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:24.796+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:24.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:24.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:24.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:24.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:24.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:24.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:24.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:24.768+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:25.791 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:25.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:25.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:25.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:25.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:25.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:25.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:25.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:25.792 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:25.766+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:25.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:25.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:25.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:25.791+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:26.748 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:26.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:26.748 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:26.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:26.749 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:26.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:26.749 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:26.484+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:26.859+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12808 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:26.747+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:26.747+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:26.747+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:26.747+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:26.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:26.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:26.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:26.803+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:27.709 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:27.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:27.710 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:27.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:27.710 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:27.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:27.710 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:27.516+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:27.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:27.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:27.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:27.756+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:27.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:27.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:27.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:27.709+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:28.748 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:28.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:28.749 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:28.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:28.749 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:28.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:28.749 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:28.493+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:28.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:28.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:28.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:28.748+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:28.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:28.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:28.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:28.750+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:29.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:29.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:29.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:29.717+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:29.540+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:29.540+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:29.540+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:29.540+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:29.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:29.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:29.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:29.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:29.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:29.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:29.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:29.762+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:30.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:30.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:30.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:30.573+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:30.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:30.703+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:30.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:30.703+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:30.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:30.703+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:30.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:30.703+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:30.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:30.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:30.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:30.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:30.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:30.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:30.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:30.771+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:31.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:31.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:31.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:31.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:31.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:31.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:31.863 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:31.701+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:31.863 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:31.747+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:31.863 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:31.747+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:31.863 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:31.747+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:31.863 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:31.747+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:31.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:31.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:31.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:31.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:31.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:31.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:31.864 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:31.584+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:32.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:31.861+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12903 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:32.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:32.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:32.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:32.600+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:32.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:32.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:32.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:32.762+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:33.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:32.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:33.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:32.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:33.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:32.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:33.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:32.744+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:33.628+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:33.628+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:33.628+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:33.629+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:33.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:33.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:33.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:33.778+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:34.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:33.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:34.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:33.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:34.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:33.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:34.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:33.786+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:34.647+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:34.647+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:34.647+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:34.647+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:34.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:34.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:34.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:34.784+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:34.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:34.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:34.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:34.759+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:35.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:35.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:35.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:35.666+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:35.770+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:35.770+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:35.770+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:35.770+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:35.724+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:35.724+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:35.724+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:35.724+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:36.989 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:36.863+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 12998 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:36.990 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:36.702+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:36.990 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:36.702+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:36.990 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:36.702+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:36.990 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:36.702+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:36.990 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:36.749+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:36.991 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:36.749+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:36.991 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:36.749+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:36.991 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:36.749+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:36.991 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:36.728+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:36.991 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:36.728+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:36.992 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:36.728+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:36.992 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:36.728+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:37.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:37.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:37.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:37.720+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:38.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:37.711+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:38.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:37.711+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:38.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:37.711+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:38.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:37.711+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:38.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:37.798+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:38.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:37.798+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:38.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:37.798+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:38.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:37.798+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:38.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:38.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:38.749+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:38.749+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:38.848+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:38.848+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:38.848+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:38.848+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:39.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:38.711+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:39.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:38.711+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:39.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:38.711+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:39.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:38.711+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:39.736+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:39.736+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:39.736+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:39.736+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:39.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:39.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:39.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:39.850+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:39.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:40.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:39.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:40.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:39.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:40.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:39.723+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:40.735+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:40.735+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:40.735+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:40.735+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:40.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:40.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:40.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:40.831+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:40.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:41.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:40.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:41.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:40.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:41.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:40.677+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:41.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:41.701+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:41.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:41.701+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:41.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:41.701+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:41.827 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:41.701+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:41.827 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:41.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:41.827 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:41.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:41.827 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:41.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:41.828 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:41.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:42.120 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:41.864+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13094 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:41.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:41.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:41.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:41.825+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:42.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:42.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:42.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:42.839+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:43.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:42.696+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:43.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:42.696+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:43.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:42.696+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:43.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:42.696+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:42.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:43.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:42.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:43.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:42.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:43.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:42.699+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:43.746+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:43.746+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:43.746+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:44.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:43.746+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:43.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:43.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:43.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:43.805+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:44.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:43.652+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:44.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:43.652+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:44.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:43.652+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:44.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:43.652+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:44.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:44.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:44.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:44.758+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:44.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:44.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:44.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:44.821+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:44.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:44.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:44.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:45.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:44.688+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:45.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:45.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:45.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:45.727+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:45.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:45.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:45.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:45.720+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:45.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:45.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:46.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:45.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:46.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:45.856+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:46.992 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:46.866+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13187 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:46.993 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:46.710+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:46.993 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:46.710+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:46.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:46.710+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:46.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:46.710+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:46.995 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:46.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:46.995 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:46.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:46.995 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:46.718+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:46.996 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:46.718+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:46.996 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:46.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:46.996 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:46.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:46.997 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:46.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:46.997 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:46.815+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:47.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:47.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:47.700+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:47.700+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:47.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:47.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:47.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:47.820+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:48.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:47.744+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:48.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:47.744+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:48.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:47.744+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:48.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:47.744+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:48.675+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:48.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:48.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:48.676+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:48.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:48.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:48.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:48.853+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:48.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:49.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:48.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:49.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:48.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:49.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:48.763+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:49.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:49.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:49.707+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:49.707+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:49.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:49.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:49.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:49.897+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:49.785+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:49.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:50.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:49.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:50.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:49.786+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:50.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:50.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:50.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:50.791+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:50.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:50.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:50.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:50.918+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:51.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:50.701+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:51.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:50.701+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:51.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:50.701+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:51.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:50.701+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:51.944 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:51.869+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13282 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:51.945 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:51.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:51.945 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:51.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:51.945 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:51.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:51.945 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:51.765+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:51.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:51.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:51.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:51.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:51.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:51.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:51.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:51.676+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:51.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:51.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:51.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:51.944+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:52.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:52.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:52.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:52.808+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:52.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:52.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:52.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:52.913+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:53.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:52.712+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:53.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:52.712+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:53.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:52.712+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:53.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:52.712+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:53.923 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:53.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:53.923 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:53.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:53.923 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:53.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:53.923 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:53.667+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:53.923 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:53.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:53.924 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:53.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:53.924 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:53.808+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:53.924 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:53.808+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:53.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:53.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:53.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:53.922+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:54.951 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:54.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:54.952 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:54.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:54.952 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:54.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:54.952 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:54.816+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:54.952 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:54.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:54.952 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:54.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:54.953 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:54.676+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:54.953 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:54.676+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:55.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:54.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:54.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:54.951+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:54.951+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:55.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:55.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:55.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:55.715+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:55.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:55.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:55.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:55.920+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:56.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:55.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:56.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:55.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:56.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:55.813+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:56.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:55.813+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:56.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:58:56.871+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13374 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:58:56.999 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:56.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:56.999 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:56.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:57.000 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:56.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:57.000 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:56.740+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:57.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:56.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:57.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:56.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:57.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:56.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:57.001 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:56.881+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:57.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:56.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:57.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:56.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:57.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:56.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:57.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:56.856+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:58.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:57.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:58.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:57.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:58.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:57.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:58.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:57.740+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:57.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:58.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:57.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:58.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:57.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:58.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:57.920+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:58.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:57.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:58.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:57.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:58.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:57.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:58.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:57.850+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:58.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:58.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:58.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:58:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:58.750+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:58.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:58.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:58.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:58:59.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:58.908+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:58:59.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:58.867+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:59.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:58.867+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:59.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:58.867+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:58:59.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:58.867+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:59.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:59.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:59.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:58:59.708+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:59.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:59.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:59.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:00.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:58:59.880+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:00.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:59.892+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:00.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:59.892+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:00.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:59.892+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:00.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:58:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:58:59.892+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:00.688+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:00.688+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:00.688+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:00.688+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:00.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:00.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:00.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:00.927+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:00.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:00.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:00.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:01.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:00.880+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:01.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:01.874+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13471 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:02.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:01.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:02.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:01.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:02.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:01.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:02.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:01.884+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:02.000 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:01.661+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:01.661+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:01.661+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:01.661+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:01.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:01.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:01.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:01.878+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:02.917 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:02.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:02.918 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:02.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:02.918 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:02.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:02.918 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:02.879+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:02.918 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:02.664+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:02.918 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:02.664+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:02.919 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:02.664+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:02.919 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:02.665+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:02.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:02.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:02.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:02.917+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:03.953 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:03.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:03.953 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:03.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:03.953 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:03.684+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:03.953 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:03.684+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:03.954 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:03.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:03.954 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:03.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:03.954 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:03.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:03.954 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:03.913+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:03.952+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:03.952+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:03.952+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:03.952+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:04.940 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:04.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:04.940 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:04.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:04.940 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:04.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:04.940 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:04.898+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:04.941 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:04.656+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:04.941 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:04.656+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:04.941 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:04.656+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:04.941 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:04.656+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:04.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:04.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:04.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:04.939+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:05.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:05.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:05.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:05.930+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:05.930+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:05.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:05.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:05.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:05.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:05.681+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:06.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:05.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:06.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:05.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:06.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:05.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:06.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:05.965+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:06.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:06.876+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13566 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:07.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:06.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:07.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:06.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:07.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:06.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:07.000 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:06.885+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:07.000 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:06.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:07.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:06.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:07.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:06.706+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:07.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:06.706+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:07.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:06.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:07.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:06.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:07.002 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:06.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:07.002 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:06.922+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:07.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:07.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:07.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:07.840+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:07.907+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:07.907+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:07.907+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:07.907+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:07.689+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:07.689+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:07.689+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:08.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:07.689+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:08.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:08.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:08.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:09.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:08.795+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:08.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:08.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:08.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:08.938+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:09.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:08.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:09.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:08.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:09.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:08.731+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:09.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:08.731+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:09.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:09.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:09.727+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:10.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:09.727+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:09.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:09.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:09.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:10.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:09.788+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:10.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:09.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:10.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:09.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:10.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:09.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:10.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:09.927+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:10.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:10.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:10.812+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:10.812+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:10.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:10.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:10.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:11.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:10.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:11.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:10.763+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:11.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:10.763+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:11.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:10.763+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:11.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:10.763+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:11.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:11.878+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13660 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:11.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:11.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:11.969 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:11.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:11.969 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:11.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:11.969 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:11.741+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:11.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:11.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:11.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:11.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:11.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:11.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:11.970 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:11.792+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:12.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:11.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:12.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:11.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:12.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:11.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:12.289 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:11.968+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:13.014 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:12.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:13.014 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:12.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:13.014 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:12.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:13.014 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:12.742+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:13.014 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:12.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:13.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:12.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:13.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:12.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:13.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:12.801+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.014+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.014+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.014+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.014+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:13.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:13.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:13.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:13.743+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:13.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:13.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:13.751+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:13.751+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:14.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:14.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:14.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:13.964+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:14.762+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:14.762+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:14.762+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:15.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:14.762+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:14.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:14.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:14.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:14.763+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:15.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:15.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:15.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:15.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:15.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:15.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:15.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:15.003+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:15.787+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:15.787+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:15.787+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:15.788+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:15.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:15.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:15.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:15.784+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:16.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:16.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:16.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:16.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:16.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:17.007 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:16.880+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13753 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:17.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:16.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:17.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:16.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:17.009 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:16.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:17.009 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:16.754+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:17.009 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:16.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:17.010 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:16.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:17.010 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:16.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:17.010 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:16.777+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:17.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:17.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:17.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:17.040+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:18.020 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:17.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:18.021 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:17.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:18.021 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:17.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:18.021 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:17.724+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:18.021 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:17.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:18.022 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:17.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:18.022 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:17.750+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:18.022 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:17.750+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.017+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.017+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:18.746+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:18.746+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:18.746+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:18.746+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:18.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:18.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:18.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:18.750+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:19.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:19.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:19.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:19.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:18.988+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:19.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:19.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:19.742+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:19.742+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:19.749+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:19.749+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:19.749+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:19.749+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:19.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:20.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:19.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:20.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:19.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:20.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:19.962+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:20.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:20.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:20.720+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:20.720+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:20.783+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:20.783+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:20.783+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:20.783+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:20.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:20.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:20.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:21.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:20.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:21.828 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:21.745+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:21.828 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:21.745+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:21.828 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:21.745+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:21.828 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:21.745+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:21.881+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13849 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:21.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:21.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:21.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:21.827+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:21.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:21.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:21.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:21.921+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:22.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:22.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:22.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:22.899+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:22.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:22.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:22.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:22.730+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:23.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:22.876+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:22.876+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:22.876+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:22.876+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:23.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:23.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:23.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:23.920+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:23.721+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:23.721+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:23.721+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:23.721+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:23.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:24.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:23.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:24.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:23.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:24.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:23.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:24.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:24.726+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:24.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:24.726+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:24.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:24.726+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:24.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:24.726+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:24.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:24.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:24.914+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:24.914+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:24.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:24.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:24.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:24.861+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:25.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:25.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:25.964+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:25.964+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:25.762+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:25.762+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:25.762+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:25.762+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:25.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:25.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:25.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:25.897+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:27.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:26.883+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 13942 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:26.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:26.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:26.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:26.905+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:26.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:26.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:26.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:26.969+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:27.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:26.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:27.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:26.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:27.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:26.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:27.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:26.774+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:27.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:27.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:27.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:27.885+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:27.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:27.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:27.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:27.796+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:27.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:27.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:28.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:27.920+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:28.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:27.920+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:28.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:28.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:28.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:28.921+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:28.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:28.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:28.759+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:28.759+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:29.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:28.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:29.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:28.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:29.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:28.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:29.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:28.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:29.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:29.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:29.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:29.913+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:29.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:29.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:29.741+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:29.741+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:29.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:30.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:29.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:30.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:29.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:30.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:29.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:30.967 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:30.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:30.967 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:30.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:30.967 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:30.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:30.967 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:30.715+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:30.968 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:30.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:30.968 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:30.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:30.968 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:30.958+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:30.968 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:30.958+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:31.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:30.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:31.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:30.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:31.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:30.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:30.966+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:31.930 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:31.886+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14037 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:31.930 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:31.678+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:31.930 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:31.678+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:31.931 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:31.678+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:31.931 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:31.679+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:31.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:31.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:31.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:31.929+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:31.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:31.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:31.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:31.933+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:32.912 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:32.642+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:32.912 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:32.642+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:32.912 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:32.642+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:32.912 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:32.642+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:32.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:32.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:32.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:32.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:32.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:32.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:32.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:32.961+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:33.991 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:33.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:33.991 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:33.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:33.992 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:33.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:33.992 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:33.639+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:33.992 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:33.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:33.992 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:33.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:33.992 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:33.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:33.992 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:33.872+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:33.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:33.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:33.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:33.991+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:35.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:34.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:35.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:34.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:35.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:34.844+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:35.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:34.844+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:35.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:34.659+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:35.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:34.659+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:35.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:34.659+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:35.042 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:34.659+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:35.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:35.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:35.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:35.040+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:36.032 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:35.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:36.033 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:35.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:36.033 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:35.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:36.033 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:35.621+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:36.033 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:35.825+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:36.033 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:35.825+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:36.033 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:35.825+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:36.034 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:35.825+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:36.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:36.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:36.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:36.032+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:36.858 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:36.582+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:36.859 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:36.582+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:36.859 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:36.582+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:36.859 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:36.582+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:36.889+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14129 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:36.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:36.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:36.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:36.858+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:37.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:37.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:37.070+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:37.070+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:37.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:37.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:37.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:37.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:37.587+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:37.884+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:38.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:37.884+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:37.884+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:37.884+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:38.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:38.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:38.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:38.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:38.027+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:38.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:38.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:38.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:38.585+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:38.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:38.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:38.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:38.931+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.021+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:39.970 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:39.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:39.970 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:39.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:39.970 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:39.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:39.971 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:39.623+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:39.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:39.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:39.969+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:40.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:39.969+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:40.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:39.982+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:40.967 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:40.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:40.967 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:40.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:40.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:40.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:40.968 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:40.653+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:40.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:40.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:40.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:40.972+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:40.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:40.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:40.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:40.967+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:41.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:41.891+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14223 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:41.971 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:41.674+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:41.971 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:41.674+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:41.971 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:41.674+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:41.971 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:41.674+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:41.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:41.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:41.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:41.973+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:41.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:41.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:41.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:42.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:41.970+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:42.999 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:42.687+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:42.999 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:42.687+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:42.999 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:42.687+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:43.000 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:42.687+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:43.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:42.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:42.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:42.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:42.998+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.012+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:43.997 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:43.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:43.997 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:43.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:43.998 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:43.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:43.998 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:43.639+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:43.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.003+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:44.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.003+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:44.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:44.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:44.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:44.601+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:44.960+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:44.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:44.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:44.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:44.960+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:45.612+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:45.612+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:45.612+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:45.612+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:45.955+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:45.955+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:45.955+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:45.955+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:45.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:45.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:45.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:45.957+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:46.895 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:46.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:46.895 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:46.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:46.895 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:46.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:46.895 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:46.630+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:47.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:46.893+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14316 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:46.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:46.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:46.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:46.985+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:46.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:46.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:46.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:47.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:46.965+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:47.942 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:47.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:47.942 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:47.666+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:47.942 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:47.667+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:47.942 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:47.667+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:48.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:48.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:48.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:48.019+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:47.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:47.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:47.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:47.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:48.947 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:48.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:48.947 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:48.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:48.947 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:48.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:48.948 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:48.639+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:49.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:49.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:49.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:49.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:49.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:49.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:49.043+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:49.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:48.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:49.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:48.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:49.374 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:48.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:49.375 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:48.946+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:49.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:49.627+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:49.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:49.627+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:49.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:49.627+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:49.966 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:49.627+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:50.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:50.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:50.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:50.077+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:49.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:49.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:49.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:49.965+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:50.924 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:50.654+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:50.925 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:50.654+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:50.925 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:50.654+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:50.925 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:50.654+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:51.123+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:51.123+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:51.123+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:51.123+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:50.923+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:50.923+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:50.923+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:50.923+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:51.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:51.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:51.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:51.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:51.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:51.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:51.897 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:51.638+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:52.156 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:51.896+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14408 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:52.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:51.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:52.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:51.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:52.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:51.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:52.157 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:51.928+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:52.600 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:52.154+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:52.600 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:52.154+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:52.600 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:52.154+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:52.601 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:52.154+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:52.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:52.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:52.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:52.600+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:53.106+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:53.106+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:53.106+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:53.106+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:52.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:52.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:52.977+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:52.977+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:53.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:53.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:53.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:53.596+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:54.134+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:54.134+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:54.134+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:54.134+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.022+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:54.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:54.554+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:54.554+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:54.554+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:54.554+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:55.167+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:55.167+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:55.167+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:55.167+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:54.991+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:55.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:55.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:55.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:55.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:55.574+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:56.213+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:56.213+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:56.213+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:56.213+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.015+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:56.588+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:56.588+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:56.588+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:56.588+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:57.184 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T06:59:56.898+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14498 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T06:59:57.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:57.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:57.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:57.185 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:56.998+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:57.606 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:57.183+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:57.606 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:57.183+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:57.607 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:57.183+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:57.607 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:57.184+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:57.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:57.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:57.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:57.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:57.606+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:58.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:58.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:58.012+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:58.012+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:58.166+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:58.166+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:58.166+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:58.166+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:58.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:58.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:58.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T06:59:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:58.588+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:59.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:59.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:59.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T06:59:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T06:59:59.038+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:59:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:59.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:59.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:59.136+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T06:59:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T06:59:59.136+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:00.080 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:59.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:00.080 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:59.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:00.080 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:59.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:00.080 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 06:59:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T06:59:59.631+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:00.080+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:00.080+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:00.080+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:00.080+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:00.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:00.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:00.116+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:00.116+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:01.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:00.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:01.057 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:00.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:01.058 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:00.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:01.058 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:00.670+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:01.056+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:01.056+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:01.056+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:01.057+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:01.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:01.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:01.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:01.115+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:01.691 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:01.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:01.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:01.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:01.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:01.653+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:01.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:01.653+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:02.027 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:01.899+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14592 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:02.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:02.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:02.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:02.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:02.077+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:02.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:02.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:02.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:02.090+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:03.097 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:02.665+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:03.097 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:02.665+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:03.097 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:02.665+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:03.098 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:02.665+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:03.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:03.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:03.095+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:03.095+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:03.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:03.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:03.121+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:03.121+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:03.661+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:03.661+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:03.661+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:03.661+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:04.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:04.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:04.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:04.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:04.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:04.139+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:04.514 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:04.139+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:04.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:04.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:04.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:04.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:04.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:04.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:04.514 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:04.145+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:04.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:04.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:04.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:04.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:04.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:04.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:04.655+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:05.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:05.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:05.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:05.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:05.163+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:05.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:05.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:05.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:05.154+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:05.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:05.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:05.638+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:05.638+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:06.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:06.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:06.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:06.144+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:06.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:06.178+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:06.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:06.178+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:06.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:06.178+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:06.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:06.178+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:06.904 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:06.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:06.904 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:06.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:06.904 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:06.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:06.904 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:06.640+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:07.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:06.902+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14684 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:07.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:07.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:07.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:07.145+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:07.146+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:07.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:07.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:07.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:07.146+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:07.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:07.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:07.613+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:07.613+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:08.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:08.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:08.180+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:08.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:08.180+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:08.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:08.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:08.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:08.099+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:08.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:08.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:08.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:08.581+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:09.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:09.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:09.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:09.187+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:09.078+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:09.078+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:09.078+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:09.079+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:09.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:09.607+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:09.607+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:09.607+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:09.607+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:10.212+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:10.212+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:10.212+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:10.213+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:10.056+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:10.056+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:10.056+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:10.056+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:10.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:10.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:10.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:10.563+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:11.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:11.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:11.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:11.014+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:11.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:11.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:11.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:11.235+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:11.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:11.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:11.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:11.539+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:12.203 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:11.905+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14777 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:12.204 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:12.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:12.204 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:12.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:12.204 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:12.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:12.204 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:12.005+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:12.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:12.202+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:12.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:12.202+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:12.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:12.202+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:12.564 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:12.203+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:12.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:12.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:12.563+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:12.563+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:13.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:13.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:13.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:13.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:13.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:13.226+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.026+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:13.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:13.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:13.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:13.537+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:14.267 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:14.267 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:14.267 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:14.268 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:13.998+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:14.498+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:14.498+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:14.498+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:14.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:14.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:14.266+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:14.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:14.266+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:14.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:14.266+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:14.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:14.266+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:15.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:14.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:15.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:14.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:15.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:14.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:15.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:14.959+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:15.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:15.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:15.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:15.458+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:15.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:15.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:15.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:15.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:15.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:15.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:15.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:15.288+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:16.247+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:16.247+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:16.247+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:16.248+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:16.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:16.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:16.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:16.008+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:16.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:16.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:16.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:16.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:16.428+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:17.244 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:16.907+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14870 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:17.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:17.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:17.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:17.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:17.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:17.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:17.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:17.044+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:17.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:17.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:17.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:17.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:17.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:17.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:17.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:17.244+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:17.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:17.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:17.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:17.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:17.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:17.438+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:17.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:17.439+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:18.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:18.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:18.065+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:18.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:18.065+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:18.215+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:18.215+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:18.215+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:18.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:18.215+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:18.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:18.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:18.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:18.480+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:19.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:19.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:19.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:19.076+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:19.245+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:19.245+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:19.245+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:19.245+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:19.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:19.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:19.471+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:19.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:19.471+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:20.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:20.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:20.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:20.045+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:20.268+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:20.268+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:20.268+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:20.268+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:20.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:20.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:20.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:20.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:21.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:21.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:21.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:21.021+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:21.260+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:21.260+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:21.260+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:21.260+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:21.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:21.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:21.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:21.525+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:22.306 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:21.910+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 14966 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:22.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:22.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:22.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:22.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:22.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:22.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:22.307 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:22.053+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:22.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:22.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:22.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:22.549+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:22.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:22.305+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:22.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:22.305+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:22.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:22.305+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:22.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:22.305+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:23.345 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:23.345 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:23.345 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.005+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:23.345 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.005+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:23.595 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:23.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:23.595 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:23.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:23.595 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:23.344+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:23.596 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:23.344+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:23.596 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:23.344+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:23.596 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:23.344+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:23.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:23.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:23.594+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:24.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:24.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:24.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:24.319 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:23.996+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:24.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:24.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:24.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:24.562+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:24.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:24.318+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:24.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:24.318+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:24.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:24.318+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:24.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:24.318+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:25.281 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:24.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:25.281 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:24.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:25.281 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:24.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:25.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:24.988+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:25.545 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:25.281+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:25.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:25.281+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:25.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:25.281+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:25.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:25.281+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:25.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:25.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:25.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:25.545+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:26.233 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:25.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:26.233 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:25.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:26.234 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:25.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:26.234 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:25.967+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:26.234 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:26.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:26.234 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:26.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:26.235 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:26.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:26.235 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:26.233+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:26.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:26.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:26.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:26.527+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:27.260 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:26.912+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15058 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:27.260 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:26.936+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:27.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:26.936+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:27.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:26.936+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:27.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:26.936+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:27.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:27.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:27.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:27.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:27.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:27.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:27.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:27.259+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:27.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:27.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:27.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:27.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:27.567+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:28.240 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:27.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:28.240 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:27.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:28.240 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:27.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:28.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:27.948+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:28.586 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:28.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:28.586 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:28.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:28.586 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:28.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:28.587 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:28.240+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:28.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:28.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:28.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:28.586+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:28.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:28.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:28.983+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:28.984+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:29.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:29.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:29.206+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:29.207+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:29.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:29.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:29.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:29.599+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:29.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:29.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:29.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:29.936+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:30.179+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:30.179+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:30.179+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:30.180+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:30.949 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:30.624+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:30.949 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:30.624+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:30.949 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:30.624+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:30.950 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:30.624+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:30.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:30.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:30.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:30.948+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:31.170+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:31.170+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:31.170+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:31.170+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:31.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:31.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:31.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:31.577+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:32.161 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:31.915+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15151 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:32.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:31.906+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:32.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:31.906+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:32.162 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:31.906+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:32.162 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:31.906+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:32.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:32.141+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:32.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:32.141+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:32.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:32.141+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:32.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:32.141+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:32.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:32.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:32.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:32.590+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:33.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:32.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:33.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:32.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:33.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:32.883+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:33.185 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:32.883+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:33.545 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:33.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:33.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:33.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:33.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:33.184+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:33.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:33.184+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:33.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:33.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:33.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:33.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:33.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:33.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:33.863 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:33.545+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:33.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:33.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:33.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:33.862+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:34.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:34.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:34.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:34.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:34.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:34.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:34.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:34.181+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:34.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:34.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:34.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:34.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:34.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:34.515+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:34.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:34.515+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:35.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:34.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:34.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:34.839+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:34.839+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:35.542 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:35.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:35.542 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:35.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:35.542 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:35.154+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:35.542 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:35.154+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:35.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:35.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:35.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:35.541+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:36.128 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:35.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:36.128 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:35.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:36.128 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:35.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:36.128 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:35.873+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:36.502 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:36.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:36.503 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:36.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:36.503 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:36.128+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:36.503 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:36.128+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:36.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:36.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:36.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:36.502+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:36.917+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15242 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:36.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:36.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:36.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:36.907+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:37.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:37.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:37.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:37.098+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:37.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:37.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:37.532+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:37.532+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:37.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:37.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:37.889+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:37.889+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:38.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:38.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:38.052+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:38.052+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:38.531+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:38.531+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:38.531+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:38.531+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:38.912+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:38.912+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:38.912+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:38.912+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:39.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:39.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:39.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:39.073+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:39.073+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:39.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:39.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:39.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:39.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:39.509+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:40.160 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:40.099+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:40.160 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:40.099+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:40.160 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:40.099+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:40.160 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:40.099+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:40.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:39.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:40.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:39.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:40.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:39.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:40.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:39.877+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:40.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:40.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:40.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:40.537+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:40.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:40.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:40.917+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:40.917+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:41.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:41.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:41.062+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:41.062+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:41.693 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:41.564+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:41.693 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:41.564+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:41.693 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:41.564+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:41.693 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:41.564+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:41.946 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:41.919+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15336 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:41.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:41.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:41.948+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:41.948+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:42.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:42.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:42.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:42.098+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:42.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:42.098+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:42.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:42.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:42.527+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:42.527+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:43.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:43.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:43.117+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:43.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:43.117+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:42.909+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:42.909+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:42.909+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:43.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:42.909+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:43.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:43.570+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:43.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:43.570+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:43.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:43.570+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:43.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:43.570+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:44.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:43.860+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:43.860+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:43.860+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:43.860+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:44.449 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:44.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:44.449 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:44.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:44.449 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:44.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:44.449 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:44.134+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:44.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:44.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:44.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:44.530+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:44.530+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:45.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:45.488 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:45.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:45.488 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:45.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:45.489 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:45.101+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:45.489 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:44.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:45.489 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:44.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:45.489 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:44.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:45.489 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:44.887+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:45.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:45.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:45.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:45.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:45.906+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:45.906+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:45.906+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:45.906+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:46.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:46.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:46.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:46.144+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:46.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:46.518+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:46.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:46.518+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:46.869 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:46.518+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:46.869 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:46.518+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:47.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:46.921+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15428 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:46.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:46.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:46.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:46.867+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:47.478 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:47.174+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:47.478 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:47.174+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:47.479 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:47.174+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:47.479 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:47.174+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:47.853 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:47.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:47.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:47.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:47.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:47.477+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:47.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:47.478+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:48.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:47.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:47.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:47.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:47.853+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:48.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:48.215+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:48.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:48.215+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:48.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:48.215+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:48.517 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:48.215+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:48.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:48.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:48.516+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:48.516+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:49.238 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:48.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:49.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:48.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:49.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:48.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:49.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:48.879+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:49.551 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:49.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:49.551 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:49.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:49.551 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:49.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:49.551 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:49.238+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:49.550+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:49.550+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:49.550+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:49.550+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:50.160 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:49.896+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:50.160 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:49.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:50.160 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:49.897+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:50.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:49.897+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:50.584 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:50.211+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:50.585 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:50.211+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:50.585 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:50.211+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:50.585 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:50.211+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:50.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:50.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:50.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:50.584+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:51.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:50.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:51.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:50.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:51.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:50.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:51.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:50.889+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:51.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:51.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:51.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:51.240+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:51.926 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:51.634+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:51.926 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:51.634+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:51.926 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:51.634+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:51.926 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:51.634+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:51.927 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:51.870+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:51.927 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:51.870+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:51.927 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:51.870+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:51.927 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:51.870+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:52.205 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:51.925+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15521 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:52.576 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:52.204+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:52.576 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:52.204+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:52.576 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:52.204+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:52.576 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:52.204+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:52.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:52.641+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:52.844 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:52.641+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:52.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:52.641+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:52.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:52.641+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:53.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:52.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:52.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:52.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:52.844+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:53.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:53.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:53.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:53.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:53.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:53.166+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:53.604 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:53.167+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:53.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:53.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:53.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:53.603+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:53.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:53.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:53.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:53.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:53.841+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:54.567 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:54.200+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:54.567 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:54.200+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:54.567 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:54.200+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:54.567 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:54.200+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:54.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:54.566+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:54.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:54.566+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:54.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:54.566+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:54.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:54.566+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:55.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:54.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:55.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:54.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:55.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:54.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:55.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:54.865+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:55.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:55.182+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:55.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:55.182+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:55.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:55.182+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:55.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:55.182+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:55.853 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:55.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:55.853 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:55.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:55.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:55.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:55.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:55.600+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:56.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:55.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:55.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:55.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:55.853+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:56.230+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:56.230+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:56.230+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:56.230+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:56.928 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:56.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:56.928 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:56.623+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:56.928 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:56.624+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:56.928 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:56.624+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:56.929 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:56.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:56.929 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:56.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:56.929 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:56.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:56.929 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:56.832+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:57.273 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:00:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:00:56.927+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15613 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:00:57.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:57.273+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:57.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:57.273+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:57.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:57.273+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:57.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:57.273+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:57.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:57.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:57.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:57.599+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:57.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:57.820+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:57.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:57.820+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:57.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:57.820+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:57.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:57.820+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:58.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:58.260+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:58.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:58.260+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:58.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:58.260+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:58.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:58.260+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:58.592+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:58.592+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:58.592+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:58.592+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:58.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:58.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:58.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:58.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:58.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:58.791+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:58.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:58.791+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:59.594 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:59.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:59.594 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:59.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:59.594 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:59.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:00:59.594 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:00:59.237+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:59.593+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:59.593+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:59.593+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:00:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:00:59.593+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:00:59.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:59.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:59.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:59.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:59.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:59.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:00:59.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:00:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:00:59.798+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:00.554 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:00.242+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:00.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:00.242+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:00.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:00.242+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:00.555 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:00.242+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:00.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:00.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:00.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:00.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:00.840 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:00.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:00.841 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:00.554+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:00.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:00.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:00.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:00.840+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:01.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:01.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:01.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:01.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:01.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:01.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:01.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:01.216+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:01.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:01.510+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:01.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:01.510+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:01.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:01.510+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:01.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:01.510+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:02.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:01.929+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15709 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:01.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:01.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:01.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:02.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:01.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:02.467+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:02.467+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:02.467+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:02.467+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:02.249+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:02.249+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:02.249+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:02.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:02.249+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:03.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:02.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:03.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:02.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:03.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:02.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:03.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:02.895+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:03.620 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:03.279+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:03.279+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:03.279+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:03.279+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:03.430+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:03.430+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:03.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:03.430+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:03.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:03.430+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:04.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:03.893+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:04.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:03.893+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:04.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:03.893+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:04.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:03.893+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:04.620 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:04.254+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:04.254+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:04.254+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:04.254+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:04.474+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:04.474+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:04.474+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:04.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:04.474+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:05.278 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:04.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:05.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:04.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:05.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:04.898+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:05.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:04.898+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:05.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:05.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:05.484+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:05.484+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:05.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:05.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:05.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:05.278+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:06.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:05.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:06.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:05.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:06.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:05.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:06.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:05.910+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:06.460+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:06.460+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:06.460+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:06.460+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:06.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:06.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:06.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:06.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:06.238+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:07.228 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:06.932+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15802 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:07.229 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:06.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:07.229 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:06.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:07.229 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:06.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:07.229 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:06.881+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:07.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:07.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:07.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:07.472+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:07.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:07.228+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:07.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:07.228+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:07.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:07.228+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:07.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:07.228+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:08.210 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:07.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:08.210 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:07.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:08.210 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:07.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:08.210 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:07.916+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:08.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:08.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:08.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:08.425+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:08.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:08.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:08.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:08.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:08.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:08.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:08.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:08.209+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:09.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:08.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:09.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:08.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:09.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:08.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:09.183 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:08.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:09.446 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:09.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:09.446 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:09.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:09.446 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:09.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:09.446 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:09.181+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:09.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:09.445+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:09.445+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:09.445+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:09.445+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:10.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:09.902+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:10.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:09.902+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:10.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:09.902+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:10.161 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:09.902+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:10.417 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:10.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:10.417 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:10.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:10.418 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:10.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:10.418 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:10.183+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:10.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:10.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:10.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:10.417+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:11.136+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:11.136+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:11.136+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:11.136+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:10.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:10.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:10.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:10.894+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:11.695 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:11.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:11.696 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:11.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:11.696 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:11.435+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:11.696 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:11.435+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:11.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:11.934+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15897 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:11.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:11.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:11.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:11.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:11.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:11.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:11.949 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:11.877+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:12.108+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:12.108+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:12.108+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:12.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:12.109+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:12.830 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:12.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:12.831 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:12.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:12.831 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:12.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:12.831 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:12.427+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:12.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:12.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:12.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:12.830+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:13.413 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:13.135+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:13.414 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:13.135+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:13.414 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:13.135+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:13.414 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:13.135+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:13.861 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:13.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:13.861 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:13.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:13.861 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:13.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:13.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:13.413+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:14.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:13.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:13.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:13.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:13.861+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:14.441 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:14.177+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:14.441 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:14.177+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:14.442 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:14.177+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:14.442 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:14.177+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:14.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:14.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:14.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:14.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:14.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:14.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:14.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:14.441+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:15.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:14.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:14.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:14.847+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:14.847+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:15.405 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:15.137+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:15.405 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:15.137+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:15.406 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:15.137+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:15.406 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:15.137+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:15.816 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:15.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:15.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:15.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:15.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:15.404+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:15.817 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:15.405+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:16.092 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:15.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:16.092 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:15.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:16.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:15.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:16.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:15.817+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:16.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:16.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:16.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:16.092+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:16.796 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:16.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:16.797 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:16.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:16.797 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:16.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:16.797 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:16.441+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:17.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:16.937+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 15989 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:17.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:16.796+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:17.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:16.796+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:17.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:16.796+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:17.063 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:16.796+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:17.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:17.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:17.114+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:17.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:17.114+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:17.811 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:17.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:17.811 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:17.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:17.812 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:17.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:17.812 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:17.480+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:18.101 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:17.811+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:18.101 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:17.811+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:18.101 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:17.811+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:18.102 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:17.811+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:18.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:18.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:18.101+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:18.101+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:18.847 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:18.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:18.847 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:18.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:18.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:18.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:18.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:18.469+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:18.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:18.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:18.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:18.847+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:19.486 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:19.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:19.487 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:19.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:19.487 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:19.144+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:19.487 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:19.145+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:19.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:19.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:19.852 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:19.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:19.852 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:19.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:19.852 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:19.486+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:20.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:19.851+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:19.851+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:19.851+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:19.851+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:20.491 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:20.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:20.491 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:20.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:20.491 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:20.122+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:20.491 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:20.122+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:20.835 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:20.491+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:20.835 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:20.491+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:20.835 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:20.491+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:20.835 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:20.491+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:21.096 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:20.835+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:21.096 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:20.835+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:21.096 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:20.835+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:21.096 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:20.835+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:21.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:21.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:21.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:21.095+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:21.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:21.456+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:21.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:21.456+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:21.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:21.456+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:21.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:21.456+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:22.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:21.938+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16085 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:21.822+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:21.822+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:21.822+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:22.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:21.822+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:22.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:22.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:22.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:22.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:22.058+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:22.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:22.058+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:22.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:22.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:22.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:22.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:22.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:22.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:22.783 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:22.429+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:23.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:22.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:23.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:22.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:23.042 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:22.782+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:23.043 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:22.782+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:23.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:23.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:23.040+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:23.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:23.040+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:23.823 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:23.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:23.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:23.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:23.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:23.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:23.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:23.425+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:24.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:24.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:24.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:24.038+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:23.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:23.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:23.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:23.819+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:24.795 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:24.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:24.795 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:24.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:24.795 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:24.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:24.796 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:24.434+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:24.794+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:24.794+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:24.794+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:24.794+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:25.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:25.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:25.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:25.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:25.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:25.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:25.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:25.038+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:25.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:25.416+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:25.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:25.416+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:25.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:25.416+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:25.772 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:25.416+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:25.771+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:25.771+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:25.771+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:25.771+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:26.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:26.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:26.020+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:26.020+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:26.740 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:26.415+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:26.740 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:26.415+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:26.740 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:26.415+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:26.741 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:26.415+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:27.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:26.942+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16177 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:27.049 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:26.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:27.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:26.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:27.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:26.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:27.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:26.739+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:27.048+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:27.048+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:27.048+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:27.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:27.048+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:27.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:27.433+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:27.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:27.433+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:27.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:27.433+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:27.767 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:27.434+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:28.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:27.765+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:28.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:27.765+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:28.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:27.765+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:28.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:27.765+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:28.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:28.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:28.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:28.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:28.037+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:28.795 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:28.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:28.796 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:28.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:28.796 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:28.426+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:28.796 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:28.427+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.002+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.002+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.002+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.002+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:28.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:28.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:28.795+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:28.795+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:29.825 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:29.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:29.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:29.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:29.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:29.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:29.826 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:29.390+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:29.989+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:29.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:29.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:29.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:29.825+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:30.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:30.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:30.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:30.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:30.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:30.417+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:30.788 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:30.417+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:30.787+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:30.787+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:30.787+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:30.787+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:30.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:30.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:30.985+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:30.985+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:31.697 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:31.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:31.698 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:31.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:31.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:31.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:31.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:31.402+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:31.953 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:31.944+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16272 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:31.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:31.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:31.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:31.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:31.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:31.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:31.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:31.774+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:32.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:31.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:31.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:31.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:31.956+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:32.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:32.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:32.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:32.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:32.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:32.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:32.822 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:32.392+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:32.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:32.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:32.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:32.821+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:32.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:32.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:32.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:32.993+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:33.853 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:33.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:33.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:33.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:33.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:33.425+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:33.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:33.425+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:33.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:33.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:33.853+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:33.853+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.018+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.018+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.018+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.018+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:34.854 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:34.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:34.855 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:34.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:34.855 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:34.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:34.855 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:34.424+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:35.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.999+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:34.999+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:34.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:34.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:34.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:34.854+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:35.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:35.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:35.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:35.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:35.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:35.424+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:35.845 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:35.425+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:35.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:35.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:35.844+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:35.844+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.022+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.022+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:36.835 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:36.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:36.836 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:36.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:36.836 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:36.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:36.836 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:36.432+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:37.121 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:36.947+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16365 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:36.834+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:36.834+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:36.834+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:37.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:36.834+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:37.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:37.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:37.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:37.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:36.993+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:37.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:37.421+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:37.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:37.421+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:37.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:37.421+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:37.819 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:37.422+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:37.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:37.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:37.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:37.819+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:38.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:38.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:38.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:38.028+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:38.838 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:38.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:38.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:38.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:38.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:38.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:38.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:38.462+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:38.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:38.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:38.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:38.838+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:39.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:39.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:39.045+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:39.045+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:39.830 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:39.470+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:39.831 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:39.470+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:39.831 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:39.470+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:39.831 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:39.471+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:39.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:39.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:39.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:39.830+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:40.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:40.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:40.031+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:40.031+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:40.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:40.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:40.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:40.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:40.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:40.469+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:40.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:40.469+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:40.808+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:40.808+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:40.808+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:40.808+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.015+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:41.823 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:41.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:41.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:41.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:41.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:41.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:41.824 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:41.496+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:42.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:41.949+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16461 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:42.075 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:41.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:42.075 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:41.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:42.076 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:41.823+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:42.076 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:41.823+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:42.076 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:42.076 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:42.076 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.990+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:42.076 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:41.990+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:42.779 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:42.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:42.780 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:42.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:42.780 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:42.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:42.780 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:42.486+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:42.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:42.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:42.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:42.779+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:42.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:42.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:42.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:42.953+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:43.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:43.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:43.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:43.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:43.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:43.458+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:43.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:43.459+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:43.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:43.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:43.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:43.743+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:43.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:43.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:43.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:43.992+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:44.703 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:44.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:44.703 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:44.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:44.704 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:44.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:44.704 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:44.411+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:45.033 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:44.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:45.033 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:44.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:45.033 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:44.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:45.033 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:44.702+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.032+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.032+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:45.436+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:45.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:45.436+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:45.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:45.436+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:45.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:45.437+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:45.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:45.660+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:45.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:45.660+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:45.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:45.660+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:45.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:45.660+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:45.993+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:46.686 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:46.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:46.687 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:46.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:46.687 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:46.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:46.687 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:46.413+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:46.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:46.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:47.013 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:46.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:47.013 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:46.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:47.013 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:46.686+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:47.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:46.952+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16554 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:47.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:47.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:47.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:47.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:47.011+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:47.707 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:47.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:47.708 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:47.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:47.708 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:47.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:47.708 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:47.455+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:48.036 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:47.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:48.037 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:47.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:48.037 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:47.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:48.037 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:47.707+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:48.036+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:48.036+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:48.036+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:48.036+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:48.722 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:48.421+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:48.723 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:48.421+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:48.723 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:48.421+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:48.723 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:48.421+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:49.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:48.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:49.051 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:48.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:49.051 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:48.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:49.051 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:48.722+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:49.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:49.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:49.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:49.050+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:49.050+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:49.678+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:49.678+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:49.678+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:49.678+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:49.447+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:49.447+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:49.447+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:49.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:49.447+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:50.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:50.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:50.081+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:50.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:50.081+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:50.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:50.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:50.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:50.640+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:50.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:50.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:50.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:50.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:50.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:50.457+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:50.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:50.457+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:51.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:51.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:51.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:51.095+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:51.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:51.095+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:51.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:51.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:51.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:51.658+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:51.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:51.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:51.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:51.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:51.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:51.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:51.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:51.485+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:52.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:51.954+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16648 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:52.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:52.131+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:52.131+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:52.131+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:52.131+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:52.874 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:52.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:52.874 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:52.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:52.874 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:52.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:52.875 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:52.641+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:52.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:52.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:52.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:52.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:52.876 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:52.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:52.876 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:52.452+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:53.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:53.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:53.092+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:53.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:53.092+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:53.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:53.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:53.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:53.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:53.645+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:53.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:53.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:53.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:53.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:53.428+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:53.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:53.428+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:54.075+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:54.075+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:54.075+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:54.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:54.075+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:54.654 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:54.383+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:54.654 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:54.383+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:54.654 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:54.383+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:54.654 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:54.383+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:55.037 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:54.654+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:55.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:54.654+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:55.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:54.654+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:55.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:54.654+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:55.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.037+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:55.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.037+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:55.691 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:55.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:55.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:55.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:55.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:55.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:55.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:55.375+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:55.989 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:55.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:55.990 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:55.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:55.990 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:55.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:55.990 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:55.690+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:56.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:56.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:56.367 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.989+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:56.368 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:55.989+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:56.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:56.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:56.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:56.367+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:56.367+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:56.959 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:56.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:56.960 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:56.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:56.960 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:56.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:56.960 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:56.653+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:57.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:01:56.957+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16741 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:01:57.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:56.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:57.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:56.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:57.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:56.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:57.334 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:56.978+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:57.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:57.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:57.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:57.333+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:57.930 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:57.675+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:57.931 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:57.675+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:57.931 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:57.675+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:57.931 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:57.675+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:58.304 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:57.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:58.304 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:57.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:58.304 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:57.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:58.305 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:57.930+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:58.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:58.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:58.304+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:58.304+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:58.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:58.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:58.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:58.702+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:58.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:58.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:58.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:01:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:58.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:59.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:59.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:59.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:59.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:59.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:59.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:01:59.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:01:59.334+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:01:59.951 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:59.699+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:59.951 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:59.699+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:59.952 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:59.699+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:01:59.952 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:01:59.699+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:00.308 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:59.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:00.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:59.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:00.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:59.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:00.309 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:01:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:01:59.951+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:00.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:00.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:00.308+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:00.308+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:00.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:00.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:00.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:00.681+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:00.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:00.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:00.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:00.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:01.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:01.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:01.291+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:01.291+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:01.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:01.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:01.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:01.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:01.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:01.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:01.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:01.655+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:02.276 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:01.959+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16837 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:02.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:01.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:02.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:01.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:02.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:01.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:02.277 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:01.913+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:02.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:02.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:02.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:02.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:02.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:02.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:02.617 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:02.275+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:02.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:02.615+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:02.615+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:02.615+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:02.615+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:03.244 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:02.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:03.245 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:02.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:03.245 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:02.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:03.245 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:02.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:03.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:03.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:03.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:03.244+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:03.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:03.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:03.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:03.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:03.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:03.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:03.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:03.652+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:04.235 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:03.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:04.235 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:03.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:04.235 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:03.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:04.235 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:03.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:04.615 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:04.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:04.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:04.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:04.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:04.234+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:04.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:04.234+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:04.614+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:04.614+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:04.614+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:04.614+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:05.207 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:04.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:05.208 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:04.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:05.208 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:04.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:05.208 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:04.922+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:05.613 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:05.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:05.613 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:05.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:05.613 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:05.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:05.613 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:05.207+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:05.612+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:05.612+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:05.612+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:05.612+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:06.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:06.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:06.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:06.178+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:05.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:05.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:05.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:05.956+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:06.600+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:06.600+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:06.600+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:06.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:06.600+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:07.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:06.963+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 16931 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:07.167+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:07.167+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:07.167+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:07.167+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:06.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:06.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:06.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:06.921+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:07.553+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:07.554+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:07.554+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:07.554+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:08.135+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:08.135+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:08.135+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:08.136+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:07.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:07.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:07.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:07.925+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:08.594+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:08.594+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:08.594+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:08.594+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:09.180 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:08.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:09.180 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:08.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:09.180 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:08.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:09.180 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:08.919+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:09.598 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:09.179+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:09.599 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:09.179+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:09.599 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:09.179+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:09.599 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:09.179+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:09.598+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:09.598+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:09.598+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:09.598+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:10.148+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:10.148+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:10.148+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:10.148+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:09.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:09.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:09.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:10.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:09.939+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:10.563+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:10.563+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:10.563+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:10.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:10.563+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:11.161 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:10.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:11.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:10.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:11.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:10.896+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:11.162 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:10.896+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:11.559 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:11.160+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:11.559 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:11.161+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:11.560 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:11.161+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:11.560 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:11.161+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:11.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:11.559+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:11.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:11.559+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:11.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:11.559+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:11.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:11.559+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:12.206 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:11.965+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17025 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:12.207 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:11.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:12.207 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:11.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:12.207 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:11.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:12.207 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:11.916+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:12.563 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:12.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:12.564 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:12.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:12.564 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:12.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:12.564 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:12.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:12.563+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:12.563+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:12.563+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:12.563+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:13.225 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:12.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:13.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:12.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:13.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:12.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:13.226 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:12.924+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:13.520 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:13.225+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:13.520 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:13.225+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:13.520 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:13.225+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:13.520 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:13.225+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:13.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:13.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:13.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:13.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:13.519+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:14.214 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:13.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:14.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:13.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:14.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:13.883+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:14.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:13.883+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:14.474 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:14.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:14.474 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:14.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:14.474 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:14.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:14.474 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:14.214+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:14.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:14.473+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:14.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:14.473+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:14.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:14.473+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:14.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:14.473+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:14.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:14.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:14.850+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:15.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:14.850+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:15.506 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:15.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:15.507 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:15.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:15.507 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:15.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:15.507 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:15.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:15.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:15.506+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:15.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:15.506+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:15.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:15.506+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:15.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:15.506+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:15.834+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:15.834+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:15.834+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:16.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:15.834+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:16.515 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:16.243+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:16.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:16.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:16.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:16.244+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:16.516 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:16.244+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:16.819 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:16.514+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:16.819 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:16.514+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:16.819 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:16.514+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:16.819 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:16.515+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:17.093 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:16.968+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17120 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:17.093 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:16.818+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:17.093 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:16.818+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:17.093 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:16.818+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:17.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:16.818+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:17.221+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:17.221+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:17.221+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:17.222+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:17.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:17.542+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:17.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:17.542+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:17.861 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:17.542+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:17.861 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:17.542+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:17.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:17.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:17.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:17.860+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:18.518 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:18.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:18.518 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:18.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:18.518 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:18.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:18.519 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:18.177+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:18.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:18.517+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:18.517+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:18.517+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:18.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:18.517+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:19.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:18.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:19.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:18.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:19.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:18.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:19.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:18.888+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:19.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:19.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:19.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:19.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:19.535 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:19.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:19.536 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:19.214+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:19.851 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:19.534+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:19.852 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:19.534+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:19.852 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:19.534+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:19.852 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:19.534+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:19.851+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:19.851+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:19.851+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:19.851+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:20.496 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:20.255+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:20.496 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:20.255+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:20.496 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:20.255+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:20.496 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:20.255+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:20.869 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:20.493+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:20.869 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:20.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:20.869 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:20.494+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:20.869 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:20.494+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:20.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:20.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:20.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:20.868+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:21.222+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:21.222+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:21.222+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:21.222+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:21.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:21.459+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:21.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:21.459+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:21.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:21.459+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:21.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:21.460+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:21.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:21.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:21.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:21.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:21.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:21.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:21.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:21.820+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:22.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:21.971+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17217 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:22.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:22.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:22.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:22.268+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:22.446+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:22.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:22.446+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:22.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:22.446+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:22.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:22.446+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:22.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:22.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:22.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:22.839+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:23.240+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:23.240+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:23.240+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:23.240+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:23.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:23.410+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:23.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:23.410+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:23.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:23.410+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:23.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:23.410+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:24.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:23.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:24.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:23.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:24.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:23.871+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:24.215 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:23.871+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:24.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:24.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:24.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:24.430+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:24.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:24.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:24.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:24.214+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:24.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:24.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:24.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:24.858+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:25.443+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:25.443+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:25.443+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:25.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:25.443+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:25.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:25.204+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:25.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:25.204+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:25.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:25.204+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:25.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:25.204+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:25.826+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:25.826+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:25.826+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:25.827+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:26.470+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:26.470+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:26.470+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:26.470+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:26.248+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:26.248+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:26.248+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:26.248+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:27.099 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:26.974+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17311 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:27.099 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:26.818+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:27.099 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:26.818+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:27.099 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:26.818+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:27.100 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:26.818+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:27.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:27.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:27.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:27.272+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:27.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:27.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:27.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:27.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:27.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:27.445+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:27.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:27.445+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:28.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:27.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:27.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:27.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:27.799+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:28.397+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:28.397+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:28.397+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:28.397+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:28.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:28.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:28.256+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:28.257+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:28.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:28.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:28.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:28.815+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:29.347+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:29.347+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:29.347+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:29.347+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:29.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:29.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:29.297+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:29.297+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:29.773+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:29.773+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:29.773+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:29.773+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:30.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:30.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:30.319+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:30.319+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:30.306+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:30.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:30.306+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:30.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:30.306+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:30.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:30.306+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:30.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:30.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:30.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:30.782+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:31.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:31.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:31.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:31.309+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:31.346+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:31.346+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:31.346+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:31.346+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:31.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:31.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:31.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:31.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:31.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:31.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:31.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:31.745+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:32.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:31.976+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17405 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:32.302+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:32.302+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:32.302+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:32.302+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:32.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:32.327+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:32.327+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:32.327+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:32.327+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:33.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:32.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:33.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:32.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:33.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:32.787+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:33.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:32.787+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:33.377 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:33.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:33.377 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:33.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:33.377 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:33.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:33.377 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:33.271+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:33.795 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:33.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:33.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:33.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:33.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:33.376+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:33.796 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:33.376+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:33.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:33.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:33.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:33.795+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:34.303+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:34.303+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:34.303+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:34.303+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:34.390+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:34.390+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:34.390+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:34.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:34.390+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:34.843+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:34.843+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:34.843+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:34.843+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:35.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:35.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:35.370+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:35.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:35.370+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:35.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:35.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:35.331+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:35.332+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:35.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:35.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:35.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:35.859+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:36.362+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:36.362+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:36.362+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:36.362+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:36.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:36.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:36.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:36.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:36.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:36.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:36.309+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:37.104 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:36.979+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17498 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:37.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:36.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:37.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:36.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:37.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:36.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:37.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:36.819+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:37.351+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:37.351+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:37.351+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:37.351+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:37.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:37.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:37.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:37.269+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:37.776+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:37.776+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:37.776+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:37.776+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:38.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:38.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:38.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:38.224+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:38.358+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:38.358+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:38.358+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:38.358+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:38.742+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:38.742+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:38.742+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:38.742+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:39.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:39.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:39.273+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:39.273+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:39.314+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:39.314+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:39.314+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:39.314+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:39.702+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:39.702+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:39.702+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:39.702+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:40.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:40.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:40.269+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:40.269+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:40.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:40.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:40.364+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:40.364+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:40.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:40.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:40.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:40.739+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:41.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:41.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:41.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:41.264+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:41.395+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:41.395+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:41.395+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:41.395+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:41.983 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:41.752+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:41.983 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:41.752+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:41.983 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:41.752+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:41.983 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:41.752+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:42.282 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:41.981+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17594 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:42.282+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:42.282+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:42.282+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:42.282+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:42.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:42.360+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:42.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:42.360+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:42.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:42.360+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:42.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:42.360+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:43.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:42.721+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:42.721+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:42.721+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:42.721+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:43.232+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:43.232+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:43.232+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:43.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:43.232+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:43.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:43.366+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:43.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:43.366+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:43.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:43.366+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:43.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:43.366+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:43.691+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:43.691+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:43.691+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:43.691+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:44.199+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:44.199+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:44.199+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:44.199+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:44.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:44.355+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:44.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:44.355+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:44.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:44.355+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:44.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:44.355+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:44.660+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:44.660+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:44.660+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:44.660+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:45.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:45.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:45.214+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:45.214+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:45.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:45.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:45.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:45.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:45.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:45.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:45.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:45.384+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:45.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:45.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:45.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:45.650+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:46.620 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:46.424+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:46.424+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:46.424+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:46.424+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:46.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:46.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:46.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:46.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:46.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:46.191+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:46.985 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:46.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:46.985 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:46.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:46.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:46.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:46.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:46.693+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:47.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:46.984+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17689 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:47.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:47.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:47.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:47.203+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:47.696 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:47.439+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:47.697 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:47.439+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:47.697 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:47.439+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:47.697 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:47.439+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:47.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:47.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:47.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:47.696+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:48.408+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:48.408+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:48.408+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:48.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:48.408+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:48.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:48.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:48.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:48.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:48.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:48.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:48.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:48.175+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:48.686+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:48.686+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:48.686+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:48.686+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:49.459 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:49.180+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:49.460 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:49.180+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:49.460 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:49.180+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:49.460 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:49.181+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:49.458+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:49.458+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:49.458+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:49.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:49.458+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:49.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:49.672+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:49.672+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:49.672+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:49.672+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:50.509 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:50.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:50.510 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:50.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:50.510 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:50.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:50.510 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:50.173+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:50.508+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:50.508+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:50.508+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:50.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:50.508+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:50.719+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:50.719+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:50.719+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:50.719+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:51.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:51.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:51.554 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:51.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:51.554 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:51.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:51.554 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:51.131+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:51.553+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:51.553+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:51.553+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:51.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:51.553+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:51.768+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:51.768+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:51.768+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:51.768+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:52.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:51.987+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17782 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:52.141+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:52.141+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:52.141+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:52.141+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:52.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:52.511+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:52.511+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:52.511+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:52.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:52.511+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:52.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:52.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:52.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:52.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:52.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:52.730+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:53.469 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:53.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:53.469 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:53.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:53.469 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:53.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:53.469 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:53.181+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:53.764 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:53.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:53.765 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:53.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:53.765 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:53.468+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:53.765 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:53.468+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:53.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:53.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:53.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:53.764+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:54.506 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:54.172+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:54.506 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:54.172+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:54.506 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:54.172+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:54.507 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:54.173+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:54.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:54.506+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:54.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:54.506+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:54.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:54.506+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:54.784 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:54.506+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:54.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:54.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:54.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:54.783+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:55.528 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:55.161+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:55.528 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:55.161+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:55.528 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:55.161+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:55.529 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:55.161+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:55.829 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:55.528+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:55.830 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:55.528+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:55.830 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:55.528+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:55.830 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:55.528+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:55.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:55.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:55.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:55.829+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:56.511 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:56.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:56.511 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:56.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:56.511 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:56.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:56.511 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:56.205+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:56.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:56.510+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:56.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:56.510+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:56.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:56.510+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:56.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:56.510+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:57.227 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:02:56.990+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17874 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:02:57.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:56.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:57.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:56.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:57.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:56.878+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:57.228 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:56.878+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:57.527 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:57.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:57.527 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:57.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:57.527 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:57.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:57.528 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:57.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:57.837 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:57.526+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:57.838 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:57.526+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:57.838 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:57.526+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:57.838 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:57.526+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:58.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:57.837+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:57.837+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:57.837+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:57.837+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:58.489 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:58.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:58.489 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:58.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:58.490 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:58.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:58.490 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:58.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:58.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:58.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:58.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:58.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:58.792 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:58.489+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:58.793 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:58.489+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:58.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:58.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:58.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:02:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:58.791+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:59.506 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:59.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:59.506 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:59.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:59.507 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:59.189+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:02:59.507 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:02:59.189+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:02:59.772 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:59.505+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:59.772 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:59.505+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:59.772 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:59.505+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:02:59.772 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:02:59.505+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:59.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:59.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:00.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:59.771+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:02:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:02:59.771+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:00.530 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:00.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:00.531 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:00.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:00.531 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:00.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:00.531 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:00.184+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:00.782 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:00.530+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:00.783 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:00.530+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:00.783 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:00.530+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:00.783 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:00.530+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:00.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:00.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:00.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:00.782+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:01.484 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:01.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:01.485 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:01.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:01.485 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:01.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:01.485 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:01.166+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:01.779 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:01.484+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:01.779 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:01.484+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:01.780 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:01.484+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:01.780 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:01.484+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:02.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:01.992+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 17969 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:02.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:01.779+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:02.118 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:01.779+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:02.118 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:01.779+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:02.118 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:01.779+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:02.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:02.141+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:02.141+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:02.141+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:02.141+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:02.817 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:02.521+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:02.817 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:02.521+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:02.817 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:02.521+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:02.817 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:02.521+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:03.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:02.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:02.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:02.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:02.816+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:03.537 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:03.172+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:03.537 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:03.172+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:03.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:03.172+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:03.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:03.172+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:03.858 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:03.536+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:03.859 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:03.536+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:03.859 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:03.536+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:03.859 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:03.537+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:03.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:03.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:03.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:03.858+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:04.583 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:04.153+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:04.584 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:04.153+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:04.584 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:04.153+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:04.584 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:04.153+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:04.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:04.583+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:04.583+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:04.583+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:04.583+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:05.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:05.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:05.143+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:05.143+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:04.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:04.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:04.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:04.900+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:05.597+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:05.597+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:05.597+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:05.597+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:06.104+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:06.104+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:06.104+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:06.104+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:05.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:05.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:05.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:05.948+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:06.941 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:06.623+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:06.941 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:06.623+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:06.941 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:06.623+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:06.942 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:06.623+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:07.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:06.995+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18063 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:07.075+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:07.075+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:07.075+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:07.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:07.075+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:06.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:06.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:06.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:06.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:07.905 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:07.637+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:07.905 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:07.637+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:07.905 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:07.637+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:07.905 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:07.637+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:08.064+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:08.064+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:08.064+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:08.064+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:07.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:07.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:07.904+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:07.904+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:08.606+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:08.606+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:08.606+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:08.606+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:08.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:08.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:08.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:08.903+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:09.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:09.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:09.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:09.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:09.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:09.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:09.085+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:10.114 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:09.650+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:10.115 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:09.650+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:10.115 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:09.650+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:10.115 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:09.650+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:10.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:09.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:10.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:09.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:10.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:09.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:10.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:09.869+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:10.114+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:10.114+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:10.114+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:10.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:10.114+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:10.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:10.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:10.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:10.695+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:10.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:10.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:10.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:10.916+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:11.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:11.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:11.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:11.154+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:11.963 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:11.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:11.964 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:11.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:11.964 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:11.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:11.964 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:11.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:11.998+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18157 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:12.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:12.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:12.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:12.154+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:11.963+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:11.963+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:11.963+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:12.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:11.963+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:13.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:12.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:12.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:12.960+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:12.961+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:13.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:12.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:13.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:12.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:13.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:12.744+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:12.744+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:13.119+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:13.119+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:13.119+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:13.119+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:14.111 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:13.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:14.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:13.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:14.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:13.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:14.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:13.748+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:14.112 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:13.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:14.112 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:13.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:14.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:13.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:14.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:13.928+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:14.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:14.111+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:14.111+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:14.111+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:14.111+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:15.082 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:14.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:15.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:14.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:15.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:14.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:15.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:14.710+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:15.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:14.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:15.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:14.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:15.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:14.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:15.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:14.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:15.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:15.082+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:15.082+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:15.082+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:15.082+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:16.046 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:15.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:16.046 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:15.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:16.046 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:15.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:16.047 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:15.710+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:16.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:15.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:16.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:15.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:16.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:15.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:16.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:15.929+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:16.046+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:16.046+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:16.046+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:16.046+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:17.002 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:16.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:17.002 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:16.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:17.003 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:16.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:17.003 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:16.743+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:17.003 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:16.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:17.003 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:16.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:17.003 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:16.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:17.003 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:16.970+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:17.001+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18251 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:17.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:17.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:17.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:17.089+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:18.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:17.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:18.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:17.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:18.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:17.940+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:18.074 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:17.940+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:18.074 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:17.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:18.074 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:17.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:18.074 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:17.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:18.074 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:17.758+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:18.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:18.073+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:18.073+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:18.073+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:18.073+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:19.100 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:18.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:19.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:18.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:19.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:18.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:19.101 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:18.968+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:19.101 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:18.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:19.101 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:18.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:19.101 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:18.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:19.102 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:18.726+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:19.099+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:19.099+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:19.099+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:19.100+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:20.111 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:19.958+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:20.111 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:19.958+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:20.111 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:19.958+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:20.111 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:19.958+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:20.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:19.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:20.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:19.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:20.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:19.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:20.112 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:19.724+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:20.110+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:20.110+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:20.110+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:20.110+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:20.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:20.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:20.681+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:20.681+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:20.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:20.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:20.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:20.929+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:21.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:21.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:21.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:21.155+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:21.907 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:21.644+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:21.908 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:21.644+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:21.908 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:21.644+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:21.908 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:21.644+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:22.168 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:22.003+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18347 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:22.168 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:21.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:22.168 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:21.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:22.169 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:21.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:22.169 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:21.906+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:22.595 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:22.166+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:22.596 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:22.166+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:22.596 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:22.166+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:22.596 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:22.167+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:22.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:22.595+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:22.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:22.595+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:22.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:22.595+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:22.862 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:22.595+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:22.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:22.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:22.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:22.860+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:23.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:23.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:23.175+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:23.175+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:24.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:23.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:24.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:23.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:24.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:23.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:24.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:23.836+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:24.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:23.632+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:24.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:23.632+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:24.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:23.632+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:24.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:23.632+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:24.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:24.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:24.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:24.152+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:24.152+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:24.663+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:24.663+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:24.663+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:24.663+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:24.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:24.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:24.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:24.848+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:25.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:25.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:25.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:25.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:25.154+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:25.679+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:25.679+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:25.679+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:25.679+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:25.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:25.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:25.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:25.829+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:26.139+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:26.139+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:26.139+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:26.140+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:27.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:26.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:27.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:26.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:27.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:26.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:27.008 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:26.672+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:27.008 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:26.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:27.009 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:26.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:27.009 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:26.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:27.009 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:26.792+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:27.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:27.007+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18441 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:27.112+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:27.112+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:27.112+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:27.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:27.112+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:28.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:27.629+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:28.094 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:27.629+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:28.094 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:27.629+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:28.094 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:27.629+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:28.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:27.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:28.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:27.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:28.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:27.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:28.095 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:27.820+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:28.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:28.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:28.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:28.092+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:29.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:28.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:29.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:28.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:29.094 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:28.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:29.094 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:28.673+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:29.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:28.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:29.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:28.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:29.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:28.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:29.094 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:28.803+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:29.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:29.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:29.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:29.093+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:29.721+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:29.721+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:29.721+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:30.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:29.721+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:29.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:29.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:29.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:29.781+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:30.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:30.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:30.126+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:30.126+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:31.096 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:30.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:31.096 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:30.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:31.097 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:30.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:31.097 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:30.743+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:31.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:30.748+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:31.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:30.748+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:31.098 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:30.748+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:31.098 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:30.748+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:31.095+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:31.095+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:31.095+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:31.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:31.095+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:32.011 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:31.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:32.011 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:31.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:32.011 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:31.725+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:32.012 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:31.725+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:32.012 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:31.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:32.012 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:31.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:32.012 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:31.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:32.012 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:31.722+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:32.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:32.010+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18535 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:32.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:32.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:32.131+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:32.131+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:32.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:32.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:32.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:32.763+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:32.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:32.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:32.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:32.697+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:33.408 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:33.180+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:33.409 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:33.180+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:33.409 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:33.180+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:33.409 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:33.180+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:33.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:33.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:33.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:33.763+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:33.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:33.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:33.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:33.688+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:34.151+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:34.151+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:34.151+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:34.151+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:34.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:34.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:34.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:34.797+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:34.658+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:34.658+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:34.658+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:34.659+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:35.122+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:35.122+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:35.122+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:35.122+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:35.753+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:35.753+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:35.753+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:35.753+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:35.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:35.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:35.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:35.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:36.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:36.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:36.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:36.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:36.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:36.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:36.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:36.140+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:37.014 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:36.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:37.014 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:36.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:37.014 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:36.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:37.014 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:36.643+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:37.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:36.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:37.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:36.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:37.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:36.763+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:37.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:36.763+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:37.013+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18629 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:37.157+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:37.157+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:37.157+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:37.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:37.157+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:37.714+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:37.714+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:37.714+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:37.714+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:37.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:37.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:37.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:37.684+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:38.162+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:38.162+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:38.162+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:38.162+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:38.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:38.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:38.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:38.711+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:38.709+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:38.709+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:38.709+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:38.709+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:39.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:39.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:39.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:39.178+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:39.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:39.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:39.717+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:39.717+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:39.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:39.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:39.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:39.712+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:40.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:40.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:40.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:40.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:40.708+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:40.708+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:40.708+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:40.708+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:40.665+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:40.665+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:40.665+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:40.665+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:41.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:41.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:41.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:41.228+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:42.017 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:41.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:42.017 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:41.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:42.017 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:41.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:42.018 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:41.714+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:42.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:41.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:42.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:41.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:42.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:41.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:42.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:41.755+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:42.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:42.016+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18725 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:42.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:42.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:42.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:42.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:42.268+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:42.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:42.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:42.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:42.757+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:42.713+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:42.713+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:42.713+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:42.713+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:43.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:43.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:43.247+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:43.247+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:43.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:43.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:43.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:43.736+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:43.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:43.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:43.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:43.752+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:44.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:44.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:44.264+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:44.264+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:44.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:44.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:44.749+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:44.749+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:44.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:44.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:44.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:44.758+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:45.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:45.298+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:45.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:45.298+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:45.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:45.298+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:45.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:45.298+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:45.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:45.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:45.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:45.764+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:45.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:46.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:45.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:46.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:45.722+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:10.243517+0000 (oldest deadline 2024-02-10T06:53:30.243517+0000) 2024-02-10T07:03:46.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:45.722+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:46.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:46.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:46.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:46.322+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:47.019 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:46.715+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:47.020 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:46.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:47.020 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:46.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:47.020 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:46.740+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:47.020 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:46.740+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:47.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:47.019+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18818 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:47.337+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:47.337+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:47.337+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:47.337+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:47.688+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:47.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:47.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:47.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:47.739+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:48.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:48.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:48.361+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:48.361+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:49.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:48.717+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:48.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:48.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:48.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:48.701+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:49.696 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:49.383+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:49.696 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:49.383+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:49.696 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:49.383+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:49.697 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:49.383+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:49.759+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:49.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:49.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:49.695+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:49.695+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:50.396+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:50.397+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:50.397+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:50.397+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:50.764+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:50.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:50.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:50.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:50.684+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:51.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:51.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:51.413+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:51.678 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:51.413+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:52.022 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:51.752+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:52.023 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:51.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:52.023 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:51.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:52.023 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:51.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:52.023 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:51.677+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:52.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:52.022+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 18912 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:52.717 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:52.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:52.718 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:52.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:52.718 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:52.371+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:52.718 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:52.371+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:53.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:52.747+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:52.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:52.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:52.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:19.747107+0000 (oldest deadline 2024-02-10T06:53:39.747107+0000) 2024-02-10T07:03:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:52.717+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:53.734 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:53.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:53.734 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:53.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:53.734 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:53.375+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:53.734 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:53.375+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:54.120 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:53.756+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:53.733+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:54.693 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:54.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:54.694 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:54.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:54.694 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:54.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:54.694 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:54.402+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:55.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:54.730+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:54.693+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:55.691 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:55.389+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:55.691 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:55.389+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:55.691 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:55.389+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:55.692 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:55.389+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:55.690+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:55.734+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:56.714 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:56.649+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:56.716 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:56.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:56.717 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:56.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:56.717 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:56.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:53:21.576560+0000 (oldest deadline 2024-02-10T06:53:41.576560+0000) 2024-02-10T07:03:56.717 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:56.410+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:57.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:56.714+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:57.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:03:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:03:57.025+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19006 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:03:57.661 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:57.427+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:57.660+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:57.673+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:58.416+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:58.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:58.641+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:58.644+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:59.680 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:03:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:03:59.400+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:03:59.681 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:03:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:03:59.592+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:00.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:03:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:03:59.680+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:00.576 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:00.357+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:00.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:00.575+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:00.713+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:01.620 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:01.545+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:01.345+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:02.028 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:01.716+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:02.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:02.026+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19102 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:02.319+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:02.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:02.521+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:03.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:02.732+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:03.538 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:03.317+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:03.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:03.538+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:03.766+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:04.567 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:04.288+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:04.567+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:04.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:04.802+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:05.616 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:05.243+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:05.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:05.818+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:05.616+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:06.208+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:07.031 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:06.866+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:07.031 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:06.662+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:07.171+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:07.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:07.030+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19196 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:08.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:07.665+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:07.909+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:08.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:08.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:08.634+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:08.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:08.634+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:08.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:08.634+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:08.949 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:08.634+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:09.224 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:08.947+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:09.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:09.223+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:09.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:09.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:09.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:09.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:09.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:09.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:09.916 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:09.663+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:10.174 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:09.915+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:10.181+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:10.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:10.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:10.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:10.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:10.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:10.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:10.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:10.651+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:11.222 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:10.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:11.222+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:12.034 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:11.881+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:12.034 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:11.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:12.035 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:11.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:12.035 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:11.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:12.035 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:11.687+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:12.372 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:12.033+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19290 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:12.181+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:13.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:12.853+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:12.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:13.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:12.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:13.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:12.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:13.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:12.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:13.208+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:14.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:13.811+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:13.734+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:13.734+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:14.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:13.734+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:14.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:13.734+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:14.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:14.215+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:15.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:14.797+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:14.721+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:14.721+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:15.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:14.721+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:14.721+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:15.222+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:16.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:15.810+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:15.728+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:15.728+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:15.728+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:16.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:15.728+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:16.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:16.215+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:17.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:16.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:17.038 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:16.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:17.038 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:16.836+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:17.038 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:16.836+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:17.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:16.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:17.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:16.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:17.038 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:16.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:17.039 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:16.686+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:17.036+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19384 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:17.212+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:17.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:17.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:17.823+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:17.824+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:17.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:17.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:17.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:17.696+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:18.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:18.181+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:18.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:18.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:18.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:18.785+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:19.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:18.647+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:18.647+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:18.647+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:18.647+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:19.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:19.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:19.224+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:19.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:19.224+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:19.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:19.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:19.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:19.736+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:19.650+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:19.650+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:19.650+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:19.650+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:20.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:20.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:20.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:20.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:20.272+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:20.763+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:20.763+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:20.763+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:20.763+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:20.678+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:20.678+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:20.678+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:20.678+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:21.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:21.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:21.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:21.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:21.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:21.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:21.756 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:21.270+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:22.040 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:21.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:22.041 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:21.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:22.041 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:21.695+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:22.041 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:21.695+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:22.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:21.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:22.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:21.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:22.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:21.797+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:22.043 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:21.797+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:22.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:22.039+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19480 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:22.223+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:22.223+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:22.223+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:22.223+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:22.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:22.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:22.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:22.655+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:22.765+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:22.765+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:22.765+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:22.765+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:23.182+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:23.182+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:23.182+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:23.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:23.182+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:23.667+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:23.667+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:23.667+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:23.667+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:23.737+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:24.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:23.737+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:24.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:23.737+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:24.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:23.737+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:24.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:24.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:24.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:24.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:24.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:24.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:24.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:25.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:24.642+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:24.774+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:24.774+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:24.774+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:24.774+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:25.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:25.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:25.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:25.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:26.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:25.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:26.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:25.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:26.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:25.803+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:26.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:25.803+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:26.124 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:25.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:26.124 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:25.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:26.124 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:25.671+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:26.124 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:25.671+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:26.266+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:26.266+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:26.266+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:26.266+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:27.044 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:26.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:27.044 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:26.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:27.045 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:26.762+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:27.045 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:26.762+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:27.045 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:26.668+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:27.045 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:26.668+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:27.046 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:26.668+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:27.046 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:26.668+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:27.302 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:27.042+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19574 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:27.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:27.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:27.301+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:27.302+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:27.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:28.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:27.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:28.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:27.672+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:28.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:27.672+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:27.758+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:27.758+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:28.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:27.758+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:28.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:27.758+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:28.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:28.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:28.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:28.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:28.322+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:28.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:28.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:28.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:28.754+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:28.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:28.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:29.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:28.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:29.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:28.720+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:29.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:29.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:29.333+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:29.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:29.333+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:29.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:29.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:29.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:29.715+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:29.674+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:29.674+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:29.674+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:29.674+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:30.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:30.340+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:30.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:30.340+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:30.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:30.340+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:30.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:30.340+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:30.753+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:30.753+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:30.753+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:30.754+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:30.715+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:30.715+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:30.715+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:30.715+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:31.707 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:31.377+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:31.707 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:31.377+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:31.707 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:31.377+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:31.707 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:31.377+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:32.046 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:31.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:32.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:31.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:32.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:31.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:32.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:31.745+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:32.047 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:31.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:32.047 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:31.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:32.047 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:31.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:32.048 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:31.706+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:32.352 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:32.045+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19668 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:32.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:32.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:32.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:32.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:32.352+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:32.741+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:32.741+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:32.741+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:32.742+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:32.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:32.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:32.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:32.777+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:33.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:33.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:33.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:33.346+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:33.346+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:33.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:33.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:33.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:34.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:33.719+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:33.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:33.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:33.759+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:33.759+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:34.344+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:34.344+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:34.344+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:34.344+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:34.761+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:34.761+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:34.761+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:34.761+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:34.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:34.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:34.767+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:34.767+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:35.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:35.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:35.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:35.369+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:35.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:35.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:35.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:35.803+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:35.811+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:35.811+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:35.811+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:35.811+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:36.792 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:36.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:36.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:36.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:36.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:36.410+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:36.793 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:36.411+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:37.049 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:36.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:37.050 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:36.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:37.050 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:36.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:37.050 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:36.792+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:37.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:36.851+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:37.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:36.851+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:37.051 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:36.851+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:37.051 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:36.851+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:37.048+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19762 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:37.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:37.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:37.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:37.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:37.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:37.441+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:37.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:37.441+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:37.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:37.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:37.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:38.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:37.838+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:37.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:37.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:37.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:37.743+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:38.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:38.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:38.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:38.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:38.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:38.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:38.820 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:38.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:38.820 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:38.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:38.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:38.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:38.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:38.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:38.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:38.699+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:38.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:38.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:38.819+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:39.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:38.819+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:39.869 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:39.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:39.869 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:39.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:39.869 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:39.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:39.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:39.696+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:39.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:39.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:39.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:39.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:39.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:39.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:39.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:39.525+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:39.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:39.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:39.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:39.867+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:40.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:40.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:40.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:40.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:40.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:40.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:40.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:40.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:40.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:40.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:40.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:40.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:40.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:40.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:40.873+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:40.873+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:41.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:41.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:41.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:41.461+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:41.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:41.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:41.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:41.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:41.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:41.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:41.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:41.679+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:42.177 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:42.050+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19858 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:42.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:41.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:42.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:41.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:42.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:41.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:42.178 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:41.914+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:42.866 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:42.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:42.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:42.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:42.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:42.507+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:42.867 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:42.507+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:42.867 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:42.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:42.867 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:42.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:42.868 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:42.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:42.868 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:42.722+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:42.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:42.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:42.865+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:42.865+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:43.859 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:43.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:43.859 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:43.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:43.860 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:43.541+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:43.860 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:43.541+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:43.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:43.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:43.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:43.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:43.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:43.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:43.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:43.699+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:43.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:43.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:43.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:43.858+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:44.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:44.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:44.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:44.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:44.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:44.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:44.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:44.827+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:44.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:44.704+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:44.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:44.704+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:44.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:44.704+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:44.874 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:44.704+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:44.874 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:44.583+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:44.874 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:44.583+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:44.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:44.583+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:44.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:44.583+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:45.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:45.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:45.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:45.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:45.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:45.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:45.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:45.799+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:45.750+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:45.750+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:45.750+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:45.750+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:45.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:45.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:45.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:45.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:45.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:45.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:45.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:45.553+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:46.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:46.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:46.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:46.569+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:46.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:46.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:46.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:46.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:46.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:46.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:46.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:46.804+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:46.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:46.737+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:46.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:46.737+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:46.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:46.737+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:46.873 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:46.737+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:47.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:47.054+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 19952 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:47.715 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:47.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:47.715 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:47.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:47.715 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:47.569+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:47.715 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:47.569+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:47.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:47.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:47.850+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:47.851+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:47.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:47.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:47.714+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:47.714+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:48.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:48.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:48.701+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:48.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:48.701+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:48.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:48.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:48.584+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:48.584+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:48.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:48.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:48.895+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:48.896+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:49.582+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:49.582+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:49.582+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:49.582+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:49.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:49.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:49.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:49.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:49.716+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:50.176 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:49.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:50.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:49.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:50.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:49.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:50.177 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:49.914+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:50.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:50.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:50.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:50.590+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:50.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:50.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:50.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:50.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:50.743+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:50.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:50.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:50.913+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:50.913+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:51.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:51.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:51.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:51.551+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:51.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:51.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:51.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:51.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:51.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:51.718+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:52.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:52.057+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20046 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:51.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:51.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:51.944+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:51.944+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:52.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:52.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:52.551+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:52.551+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:52.673+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:52.673+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:52.673+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:52.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:52.673+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:53.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:52.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:53.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:52.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:53.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:52.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:53.282 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:52.906+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:53.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:53.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:53.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:53.588+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:53.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:53.640+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:53.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:53.640+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:53.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:53.640+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:53.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:53.640+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:54.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:53.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:53.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:53.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:53.935+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:54.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:54.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:54.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:54.543+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:54.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:54.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:54.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:54.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:54.613+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:54.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:54.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:54.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:54.981+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:55.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:55.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:55.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:55.549+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:55.584+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:55.584+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:55.584+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:55.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:55.584+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.009+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:56.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:56.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:56.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:56.529+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:56.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:56.558+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:56.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:56.558+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:56.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:56.558+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:56.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:56.558+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:57.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:04:57.060+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20140 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:04:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.981+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:56.981+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:57.605+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:57.605+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:57.605+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:57.605+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:57.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:57.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:57.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:57.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:58.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:58.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:58.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:58.021+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:58.874 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:58.607+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:58.874 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:58.607+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:58.874 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:58.607+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:58.874 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:58.607+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:58.874 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:58.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:58.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:58.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:58.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:58.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:58.875 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:58.477+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:59.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:59.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:59.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:04:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:04:59.019+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:59.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:59.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:59.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:04:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:04:59.642+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:04:59.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:59.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:59.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:59.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:59.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:59.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:04:59.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:04:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:04:59.442+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.010+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.010+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.010+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.010+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:00.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:00.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:00.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:00.455+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:00.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:00.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:00.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:00.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:00.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:00.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:00.601+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:00.992+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:01.710 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:01.609+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:01.711 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:01.609+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:01.711 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:01.609+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:01.711 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:01.609+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:01.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:01.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:01.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:01.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:01.712 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:01.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:01.713 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:01.411+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:02.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:02.063+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20238 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:02.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:02.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:02.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:02.031+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:02.654 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:02.385+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:02.655 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:02.385+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:02.655 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:02.385+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:02.655 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:02.386+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:03.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:02.654+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:03.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:02.654+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:03.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:02.654+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:03.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:02.654+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:03.352 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:03.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:03.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:03.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:03.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:03.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:03.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:03.052+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:03.613 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:03.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:03.613 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:03.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:03.614 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:03.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:03.614 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:03.352+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:03.612+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:03.612+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:03.612+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:03.613+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:04.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:04.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:04.099+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:04.099+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:04.651 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:04.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:04.651 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:04.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:04.651 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:04.390+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:04.651 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:04.390+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:05.091 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:04.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:05.091 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:04.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:05.091 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:04.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:05.091 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:04.650+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:05.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:05.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:05.090+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:05.090+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:05.682 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:05.430+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:05.682 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:05.430+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:05.682 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:05.430+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:05.682 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:05.430+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:05.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:05.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:05.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:05.681+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:06.446 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:06.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:06.446 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:06.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:06.447 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:06.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:06.447 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:06.131+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:06.446+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:06.446+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:06.446+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:06.446+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:06.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:06.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:06.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:06.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:06.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:06.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:06.689+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:07.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:07.067+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20332 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:07.088+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:07.088+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:07.088+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:07.088+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:07.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:07.416+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:07.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:07.416+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:07.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:07.416+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:07.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:07.416+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:08.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:07.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:08.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:07.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:08.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:07.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:08.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:07.699+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:08.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:08.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:08.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:08.052+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:08.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:08.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:08.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:08.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:08.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:08.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:08.681 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:08.400+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:09.071 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:08.678+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:09.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:08.678+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:09.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:08.678+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:09.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:08.678+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:09.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:09.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:09.353 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:09.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:09.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:09.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:09.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:09.071+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:09.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:09.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:09.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:09.352+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:09.353+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:10.043 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:09.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:10.044 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:09.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:10.044 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:09.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:10.044 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:09.643+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:10.369 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:10.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:10.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:10.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:10.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:10.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:10.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:10.043+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:10.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:10.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:10.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:10.369+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:10.369+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:11.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:10.666+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:11.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:10.666+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:11.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:10.666+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:11.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:10.666+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:11.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:11.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:11.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:11.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:11.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:11.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:11.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:11.072+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:11.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:11.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:11.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:11.360+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:11.360+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:12.046 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:11.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:12.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:11.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:12.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:11.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:12.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:11.689+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:12.070+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20427 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:12.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:12.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:12.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:12.046+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:12.672 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:12.381+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:12.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:12.381+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:12.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:12.381+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:12.673 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:12.381+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:13.034 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:12.672+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:13.034 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:12.672+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:13.034 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:12.672+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:13.034 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:12.672+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:13.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:13.033+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:13.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:13.033+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:13.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:13.033+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:13.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:13.033+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:13.354+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:13.354+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:13.354+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:13.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:13.354+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:14.009 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:13.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:14.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:13.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:14.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:13.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:14.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:13.683+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:14.317 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:14.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:14.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:14.318 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.009+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:14.317+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:14.317+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:14.317+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:14.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:14.317+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:14.988 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:14.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:14.989 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:14.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:14.989 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:14.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:14.989 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:14.696+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:15.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:15.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:15.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:15.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:14.988+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:15.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:15.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:15.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:15.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:15.311+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:15.967 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:15.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:15.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:15.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:15.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:15.708+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:15.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:15.708+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:16.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:15.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:16.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:15.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:16.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:15.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:16.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:15.967+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:16.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:16.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:16.261+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:16.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:16.261+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:16.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:16.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:16.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:16.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:16.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:16.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:16.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:16.677+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:17.272 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:17.073+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20521 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:17.272 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:16.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:17.272 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:16.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:17.272 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:16.968+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:17.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:16.968+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:17.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:17.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:17.272+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:17.272+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:18.013 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:17.706+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:18.013 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:17.706+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:18.014 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:17.706+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:18.014 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:17.707+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:18.268 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:18.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:18.268 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:18.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:18.268 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:18.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:18.268 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:18.013+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:18.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:18.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:18.268+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:18.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:18.268+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:19.012 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:18.746+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:19.013 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:18.746+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:19.013 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:18.746+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:19.013 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:18.746+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:19.260+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:19.260+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:19.260+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:19.260+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:19.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:19.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:19.012+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:19.012+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:20.054 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:19.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:20.055 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:19.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:20.055 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:19.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:20.055 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:19.786+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:20.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:20.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:20.286+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:20.286+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:20.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:20.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:20.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:20.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:20.053+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:21.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:20.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:20.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:20.791+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:21.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:20.791+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:21.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:21.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:21.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:21.025+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:21.318+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:21.318+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:21.318+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:21.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:21.318+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:22.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:21.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:22.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:21.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:22.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:21.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:22.073 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:21.754+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:22.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:22.075+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20619 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:22.327 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:22.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:22.327 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:22.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:22.327 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:22.072+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:22.327 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:22.072+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:22.325+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:22.325+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:22.325+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:22.325+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:23.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:22.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:23.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:22.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:23.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:22.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:23.103 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:22.754+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:23.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:23.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:23.102+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:23.102+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:23.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:23.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:23.300+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:23.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:23.300+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:23.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:23.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:23.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:23.751+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:24.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:24.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:24.152+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:24.152+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:24.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:24.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:24.283+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:24.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:24.283+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:24.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:24.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:24.715+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:24.715+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:25.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:25.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:25.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:25.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:25.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:25.199+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:25.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:25.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:25.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:25.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:25.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:25.311+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:25.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:25.311+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:25.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:25.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:25.727+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:25.727+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:26.209+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:26.209+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:26.209+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:26.210+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:26.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:26.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:26.271+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:26.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:26.271+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:27.082 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:26.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:27.082 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:26.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:27.082 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:26.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:27.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:26.686+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:27.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:27.079+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20713 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:27.196+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:27.196+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:27.196+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:27.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:27.196+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:27.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:27.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:27.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:27.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:27.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:27.233+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:27.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:27.233+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:28.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:27.657+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:28.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:27.657+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:28.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:27.657+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:28.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:27.657+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:28.159+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:28.159+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:28.159+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:28.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:28.160+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:28.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:28.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:28.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:28.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:28.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:28.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:28.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:28.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:28.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:28.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:28.682+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:29.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:29.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:29.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:29.164+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:29.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:29.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:29.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:29.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:29.220+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:29.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:29.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:29.699+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:29.699+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:30.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:30.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:30.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:30.156+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:30.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:30.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:30.239+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:30.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:30.240+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:30.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:30.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:30.739+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:31.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:30.739+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:31.179+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:31.179+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:31.179+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:31.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:31.179+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:31.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:31.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:31.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:31.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:31.201+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:32.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:31.698+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:32.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:31.698+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:32.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:31.698+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:32.083 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:31.699+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:32.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:32.082+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20808 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:32.177+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:32.177+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:32.177+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:32.177+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:32.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:32.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:32.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:32.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:32.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:32.220+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:32.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:32.220+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:32.655+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:32.655+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:32.655+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:32.655+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:33.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:33.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:33.257+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:33.257+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:33.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:33.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:33.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:33.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:33.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:33.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:33.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:33.206+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:34.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:33.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:33.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:33.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:34.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:33.643+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:34.194+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:34.194+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:34.194+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:34.194+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:34.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:34.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:34.259+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:34.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:34.260+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:35.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:34.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:34.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:34.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:34.679+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:35.246+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:35.246+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:35.246+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:35.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:35.246+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:35.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:35.212+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:35.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:35.212+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:35.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:35.212+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:35.623 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:35.212+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:35.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:35.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:35.682+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:35.682+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:36.249+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:36.249+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:36.249+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:36.249+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:36.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:36.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:36.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:36.246+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:37.086 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:36.666+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:37.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:36.666+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:37.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:36.666+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:37.087 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:36.666+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:37.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:37.085+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 20902 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:37.269+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:37.269+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:37.269+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:37.269+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:37.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:37.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:37.217+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:37.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:37.217+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:37.669+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:37.669+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:37.669+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:37.670+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:38.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:38.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:38.213+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:38.213+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:38.220+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:38.220+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:38.220+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:38.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:38.220+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:38.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:38.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:38.717+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:38.717+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:39.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:39.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:39.178+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:39.178+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:39.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:39.197+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:39.197+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:39.197+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:39.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:39.197+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:39.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:39.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:39.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:39.723+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:40.194+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:40.194+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:40.194+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:40.194+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:40.159+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:40.159+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:40.159+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:40.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:40.159+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:40.770+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:40.770+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:40.770+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:40.770+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:41.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:41.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:41.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:41.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:41.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:41.201+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:41.153+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:41.153+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:41.153+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:41.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:41.153+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:42.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:41.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:42.090 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:41.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:42.090 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:41.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:42.090 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:41.730+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:42.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:42.089+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21000 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:42.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:42.160+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:42.160+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:42.160+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:42.153+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:42.153+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:42.153+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:42.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:42.153+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:43.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:42.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:43.116 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:42.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:43.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:42.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:43.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:42.745+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:43.115+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:43.115+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:43.115+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:43.115+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:43.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:43.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:43.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:43.155+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:44.074 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:43.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:44.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:43.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:44.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:43.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:44.075 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:43.697+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:44.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:44.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:44.115+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:44.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:44.115+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:44.074+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:44.074+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:44.074+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:44.074+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:45.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:44.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:45.041 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:44.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:45.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:44.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:45.042 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:44.692+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:45.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:45.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:45.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:45.156+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.041+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.041+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.041+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:45.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.041+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:45.996 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:45.658+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:45.996 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:45.658+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:45.996 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:45.658+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:45.996 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:45.658+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.996+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.996+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.996+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:45.996+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:46.189+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:46.189+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:46.189+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:46.189+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:46.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:46.702+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:46.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:46.702+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:46.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:46.702+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:46.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:46.702+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:47.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:47.092+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21091 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:46.985+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:46.985+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:46.985+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:47.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:46.985+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:47.158+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:47.158+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:47.158+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:47.158+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:48.007 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:47.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:48.007 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:47.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:48.008 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:47.745+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:48.008 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:47.745+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.006+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.006+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.006+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.007+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:48.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:48.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:48.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:48.207+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:48.993 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:48.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:48.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:48.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:48.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:48.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:48.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:48.712+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:49.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:48.993+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:49.169+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:49.169+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:49.169+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:49.169+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:49.971+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:49.971+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:49.971+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:49.971+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:49.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:49.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:49.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:49.732+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:50.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:50.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:50.164+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:50.164+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:51.000 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:50.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:51.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:50.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:51.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:50.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:51.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:50.743+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:51.180+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:51.180+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:51.180+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:51.180+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:51.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:51.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:51.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:51.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:51.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:51.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:51.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:51.000+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:52.051 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:51.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:52.051 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:51.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:52.051 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:51.720+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:52.051 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:51.720+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:52.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:52.095+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21183 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:52.050+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:52.050+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:52.050+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:52.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:52.050+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:52.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:52.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:52.206+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:52.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:52.206+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:53.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:52.670+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:53.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:52.670+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:53.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:52.670+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:53.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:52.671+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:53.020+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:53.020+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:53.020+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:53.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:53.020+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:53.214+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:53.214+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:53.214+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:53.214+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:54.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:53.654+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:54.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:53.654+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:54.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:53.654+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:54.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:53.654+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:54.053+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:54.053+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:54.053+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:54.053+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:54.219+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:54.219+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:54.219+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:54.219+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:55.005 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:54.649+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:55.005 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:54.649+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:55.005 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:54.649+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:55.005 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:54.649+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:55.005+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:55.005+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:55.005+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:55.005+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:55.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:55.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:55.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:55.202+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:55.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:55.607+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:55.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:55.607+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:55.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:55.607+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:55.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:55.607+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.030+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.030+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.030+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.030+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:56.241+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:56.241+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:56.241+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:56.241+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:56.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:56.558+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:56.558+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:56.558+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:56.558+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:57.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:05:57.099+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21268 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:05:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:57.228+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:57.228+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:57.228+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:57.228+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.986+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.986+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.986+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:57.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:56.986+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:57.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:57.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:57.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:57.541+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:58.191+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:58.191+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:58.191+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:58.191+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:58.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:57.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:58.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:57.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:58.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:57.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:58.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:57.999+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:58.527+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:58.527+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:58.527+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:58.527+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:59.220 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:58.956+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:59.221 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:58.956+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:59.221 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:58.956+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:05:59.221 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:58.956+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:59.517 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:59.219+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:59.517 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:59.219+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:59.517 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:59.219+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:05:59.517 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:05:59.219+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:05:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:59.516+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:59.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:59.516+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:59.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:59.516+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:05:59.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:05:59.516+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:00.181 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:59.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:00.182 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:59.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:00.182 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:59.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:00.182 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:05:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:05:59.927+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:00.549 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:00.258+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:00.549 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:00.258+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:00.549 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:00.258+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:00.549 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:00.258+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:00.865 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:00.548+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:00.865 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:00.548+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:00.865 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:00.548+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:00.865 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:00.548+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:00.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:00.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:00.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:00.909+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:01.284+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:01.284+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:01.284+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:01.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:01.284+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:01.532+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:01.532+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:01.532+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:01.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:01.532+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:02.103 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:01.945+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:02.103 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:01.945+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:02.104 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:01.945+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:02.104 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:01.945+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:02.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:02.101+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21357 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:02.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:02.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:02.246+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:02.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:02.246+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:02.579+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:02.579+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:02.579+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:02.579+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:03.204 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:02.948+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:03.205 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:02.948+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:03.205 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:02.948+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:03.205 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:02.948+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:03.610 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:03.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:03.611 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:03.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:03.611 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:03.202+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:03.611 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:03.203+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:03.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:03.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:03.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:03.610+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:04.177+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:04.177+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:04.177+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:04.177+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:04.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:03.954+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:04.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:03.954+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:04.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:03.954+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:04.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:03.954+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:04.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:04.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:04.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:04.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:04.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:04.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:04.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:04.570+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:05.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:05.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:05.143+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:05.143+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:04.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:05.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:04.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:05.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:04.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:05.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:04.909+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:05.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:05.540+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:05.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:05.540+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:05.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:05.540+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:05.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:05.540+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:06.093+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:06.093+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:06.093+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:06.094+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:05.931+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:05.931+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:05.931+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:06.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:05.932+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:06.509+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:06.509+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:06.509+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:06.509+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:07.230 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:07.105+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21442 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:07.231 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:07.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:07.231 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:07.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:07.231 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:07.080+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:07.231 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:07.080+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:07.231 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:06.963+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:07.232 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:06.963+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:07.232 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:06.963+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:07.232 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:06.963+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:07.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:07.529+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:07.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:07.529+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:07.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:07.529+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:07.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:07.529+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:07.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:07.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:07.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:07.960+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:08.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:08.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:08.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:08.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:08.041+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:08.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:08.551+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:08.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:08.551+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:08.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:08.551+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:08.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:08.551+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:09.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:09.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:09.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:09.000+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:09.049+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:09.049+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:09.049+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:09.049+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:09.568+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:09.568+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:09.568+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:09.568+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:10.003+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:10.003+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:10.003+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:10.003+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:10.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:10.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:10.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:10.091+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:10.604+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:10.604+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:10.604+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:10.604+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.026+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:11.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.026+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:11.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.026+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:11.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.026+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:11.105+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:11.105+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:11.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:11.105+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:11.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:11.105+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:11.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:11.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:11.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:11.594+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:12.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:12.108+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21532 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:12.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:12.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:12.077+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:12.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:12.078+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:11.999+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:12.993 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:12.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:12.993 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:12.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:12.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:12.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:12.994 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:12.632+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:12.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:12.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:12.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:12.993+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:13.074+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:13.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:13.074+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:13.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:13.074+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:13.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:13.074+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:13.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:13.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:13.594+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:13.594+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:14.035+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:14.035+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:14.035+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:14.035+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:14.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.026+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:14.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:14.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:14.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:14.564+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:15.066+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:15.066+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:15.066+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:15.066+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:14.989+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:15.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:15.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:15.564+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:15.565+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:16.031+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:16.031+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:16.031+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:16.031+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:16.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:16.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:16.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:16.004+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:16.606+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:16.606+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:16.606+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:16.607+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:17.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:17.111+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21623 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:17.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:17.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:17.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:17.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:17.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:17.053+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:17.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:17.053+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:17.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:17.053+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:17.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:17.053+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:18.007 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:17.638+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:18.007 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:17.638+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:18.007 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:17.638+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:18.007 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:17.638+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.006+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.006+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.006+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.006+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:18.083+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:18.083+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:18.083+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:18.083+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:18.601+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:18.601+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:18.601+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:18.602+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.967+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:18.967+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:19.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:19.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:19.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:19.079+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:19.576+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:19.576+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:19.576+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:19.576+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:19.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:19.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:19.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:19.957+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:20.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:20.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:20.092+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:20.092+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:20.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:20.537+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:20.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:20.537+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:20.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:20.537+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:20.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:20.537+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:20.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:20.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:20.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:20.929+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:21.088+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:21.088+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:21.088+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:21.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:21.088+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:21.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:21.581+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:21.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:21.581+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:21.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:21.581+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:21.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:21.581+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:22.240 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:22.115+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21721 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:21.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:21.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:21.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:22.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:21.965+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:22.242 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:22.038+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:22.242 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:22.038+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:22.242 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:22.038+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:22.242 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:22.038+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:22.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:22.546+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:22.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:22.546+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:22.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:22.546+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:22.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:22.546+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:23.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:23.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:23.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:23.063+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:22.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:22.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:22.964+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:22.964+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:23.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:23.560+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:23.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:23.560+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:23.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:23.560+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:23.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:23.560+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.019+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.019+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.019+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.019+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:23.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:23.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:23.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:23.919+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:24.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:24.524+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:24.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:24.524+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:24.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:24.524+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:24.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:24.524+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:24.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:24.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:24.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:24.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:25.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.982+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:25.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.982+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:25.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.982+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:25.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:24.982+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:25.515+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:25.515+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:25.515+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:25.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:25.515+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:25.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:25.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:25.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:25.970+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:25.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:25.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:25.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:25.940+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:26.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:26.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:26.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:26.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:26.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:26.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:26.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:26.541+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:27.244 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:27.118+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21815 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:27.244 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:26.984+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:27.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:26.984+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:27.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:26.984+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:27.245 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:26.985+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:27.245 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:26.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:27.245 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:26.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:27.246 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:26.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:27.246 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:26.936+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:27.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:27.540+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:27.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:27.540+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:27.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:27.540+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:27.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:27.540+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:27.960+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:27.960+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:27.960+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:27.960+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:27.982+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:27.982+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:27.982+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:27.982+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:28.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:28.557+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:28.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:28.557+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:28.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:28.557+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:28.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:28.557+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.005+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.005+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:29.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.005+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:29.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.005+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:29.016+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:29.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:29.016+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:29.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:29.016+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:29.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:29.016+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:29.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:29.570+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:29.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:29.570+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:29.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:29.570+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:29.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:29.570+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.976+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.976+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.976+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:29.976+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:30.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:30.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:30.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:30.009+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:30.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:30.544+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:30.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:30.544+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:30.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:30.544+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:30.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:30.544+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:31.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:31.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:31.013+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:31.013+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:31.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:30.980+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:31.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:30.980+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:31.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:30.980+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:31.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:30.980+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:31.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:31.511+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:31.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:31.511+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:31.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:31.511+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:31.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:31.511+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:32.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:32.121+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 21909 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:31.997+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:31.997+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:31.997+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:31.997+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:32.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:32.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:32.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:32.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:32.045+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:32.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:32.523+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:32.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:32.523+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:32.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:32.523+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:32.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:32.523+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:32.961+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:32.961+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:32.961+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:32.961+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:33.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.024+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:33.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:33.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:33.569+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:33.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:33.569+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:33.947+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:33.947+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:33.947+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:33.947+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:33.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:34.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:34.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:34.541+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:34.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:34.541+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:34.965+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:34.965+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:34.965+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:34.965+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:35.011+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:35.011+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:35.011+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:35.012+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:35.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:35.495+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:35.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:35.495+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:35.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:35.495+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:35.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:35.495+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:36.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:36.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:36.009+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:36.010+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:35.948+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:35.948+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:36.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:35.948+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:36.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:35.948+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:36.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:36.525+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:36.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:36.525+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:36.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:36.525+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:36.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:36.525+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:37.249 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:37.125+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22003 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:37.249 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:37.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:37.249 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:37.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:37.250 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:37.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:37.250 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:37.032+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:37.250 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:36.978+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:37.250 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:36.978+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:37.251 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:36.978+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:37.251 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:36.978+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:37.551+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:37.551+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:37.551+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:37.551+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:37.981+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:37.981+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:37.981+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:37.981+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:38.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:38.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:38.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:38.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:38.024+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:38.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:38.539+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:38.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:38.539+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:38.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:38.539+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:38.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:38.539+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:39.003+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:39.003+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:39.003+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:39.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:39.003+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:39.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:39.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:39.035+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:39.035+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:39.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:39.575+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:39.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:39.575+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:39.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:39.575+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:39.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:39.575+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:40.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:40.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:40.031+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:40.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:40.031+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.034+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.034+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.034+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:40.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.034+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:40.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:40.599+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:40.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:40.599+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:40.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:40.599+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:40.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:40.599+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:41.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:41.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:41.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:41.020+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:41.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:41.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:41.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.999+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:41.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:40.999+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:41.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:41.602+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:41.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:41.602+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:41.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:41.602+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:41.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:41.602+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:42.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:42.127+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22101 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:42.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:42.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:42.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:42.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:42.060+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.013+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.013+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.013+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.013+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:42.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:42.604+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:42.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:42.604+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:42.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:42.604+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:42.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:42.604+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.973+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.973+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.973+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:43.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:42.973+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:43.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:43.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:43.083+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:43.083+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:43.593+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:43.593+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:43.593+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:43.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:43.593+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:44.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:44.001+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:44.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:44.001+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:44.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:44.001+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:44.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:44.001+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:44.039+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:44.039+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:44.039+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:44.039+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:44.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:44.591+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:44.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:44.591+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:44.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:44.591+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:44.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:44.591+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:45.037+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:45.037+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:45.037+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:45.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:45.038+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:45.079+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:45.079+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:45.079+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:45.079+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:45.581+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:45.581+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:45.581+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:45.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:45.581+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:46.032+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:46.032+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:46.032+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:46.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:46.032+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:46.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:46.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:46.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:46.087+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:46.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:46.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:46.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:46.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:46.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:46.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:46.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:46.600+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:47.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:47.132+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22195 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:47.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:47.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:47.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:47.042+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:47.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:47.042+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:47.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:47.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:47.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:47.048+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:48.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:47.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:48.052 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:47.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:48.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:47.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:48.053 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:47.647+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:48.054+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:48.054+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:48.054+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:48.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:48.054+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:48.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:48.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:48.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:48.052+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:49.024 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:48.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:49.024 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:48.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:49.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:48.677+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:49.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:48.678+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:49.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:49.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:49.057+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:49.057+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:49.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.023+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:49.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.023+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:49.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.023+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:49.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.024+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:49.984 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:49.674+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:49.985 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:49.674+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:49.985 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:49.674+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:49.985 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:49.674+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:50.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:50.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:50.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:50.092+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:50.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:50.092+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:50.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.980+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:50.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.980+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:50.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.980+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:50.373 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:49.980+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:50.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:50.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:50.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:50.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:50.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:50.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:50.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:50.697+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:50.971+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:50.971+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:50.971+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:51.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:50.971+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:51.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:51.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:51.119+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:51.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:51.119+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:51.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:51.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:51.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:51.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:51.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:51.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:51.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:51.664+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:52.262 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:52.135+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22291 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:52.263 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:51.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:52.263 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:51.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:52.263 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:51.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:52.263 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:51.998+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:52.263 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:52.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:52.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:52.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:52.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:52.146+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:52.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:52.146+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:52.956 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:52.674+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:52.957 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:52.674+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:52.957 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:52.674+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:52.957 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:52.675+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:53.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:53.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:53.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:53.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:53.156+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:53.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:53.157+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:53.374 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:52.955+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:53.374 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:52.955+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:53.375 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:52.955+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:53.375 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:52.955+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:53.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:53.711+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:53.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:53.711+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:53.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:53.711+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:53.969 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:53.711+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:53.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:53.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:53.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:54.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:53.967+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:54.140+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:54.140+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:54.140+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:54.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:54.140+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:54.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:54.662+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:54.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:54.662+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:54.999 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:54.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:55.000 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:54.663+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:54.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:54.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:55.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:54.998+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:55.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:54.998+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:55.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:55.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:55.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:55.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:55.155+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:55.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:55.155+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:55.958 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:55.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:55.959 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:55.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:55.959 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:55.630+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:55.959 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:55.630+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:55.958+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:55.958+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:55.958+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:56.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:55.958+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:56.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:56.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:56.113+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:56.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:56.113+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:56.582+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:56.582+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:56.582+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:56.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:56.582+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:57.263 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:06:57.139+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22385 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:06:57.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:56.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:57.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:56.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:57.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:56.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:57.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:56.927+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:57.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:57.126+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:57.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:57.127+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:57.265 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:57.127+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:57.266 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:57.127+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:57.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:57.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:57.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:57.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:57.620+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:57.961+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:57.961+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:57.961+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:58.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:57.961+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:58.094+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:58.094+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:58.094+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:58.094+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:58.592+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:58.592+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:58.592+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:58.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:58.592+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:59.274 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:58.975+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:59.274 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:58.975+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:59.274 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:58.975+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:06:59.274 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:58.975+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:59.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:59.089+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:59.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:59.089+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:59.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:59.089+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:06:59.275 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:06:59.089+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:06:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:59.554+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:59.554+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:59.554+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:06:59.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:06:59.554+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:00.082+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:00.082+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:00.082+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:00.082+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:00.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:59.950+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:00.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:59.950+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:00.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:59.950+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:00.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:06:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:06:59.950+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:00.518+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:00.518+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:00.518+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:00.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:00.518+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:01.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:01.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:01.053+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:01.053+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:01.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:00.952+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:00.952+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:00.952+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:01.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:00.952+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:01.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:01.559+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:01.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:01.559+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:01.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:01.559+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:01.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:01.559+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:02.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:02.142+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22482 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:02.269 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:02.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:02.269 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:02.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:02.269 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:02.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:02.269 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:02.054+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:02.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:01.972+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:02.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:01.972+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:02.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:01.972+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:02.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:01.972+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:02.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:02.575+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:02.575+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:02.575+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:02.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:02.575+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:02.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:02.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:02.992+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:02.992+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:03.100+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:03.100+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:03.100+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:03.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:03.100+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:03.608+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:03.608+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:03.608+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:03.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:03.608+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:04.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:04.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:04.131+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:04.131+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:04.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:03.984+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:04.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:03.984+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:04.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:03.984+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:04.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:03.984+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:05.005 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:04.641+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:05.005 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:04.641+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:05.006 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:04.641+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:05.006 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:04.641+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.004+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.004+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.004+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.004+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:05.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:05.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:05.109+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:05.109+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:05.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:05.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:05.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:05.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:05.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:05.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:05.972 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:05.642+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:05.970+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:06.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:06.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:06.091+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:06.091+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:06.944 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:06.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:06.944 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:06.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:06.944 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:06.685+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:06.945 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:06.685+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:07.272 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:07.146+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22575 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:07.272 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:06.944+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:07.272 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:06.944+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:07.273 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:06.944+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:07.273 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:06.944+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:07.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:07.121+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:07.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:07.121+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:07.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:07.121+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:07.273 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:07.121+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:07.912+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:07.912+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:07.912+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:07.912+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:07.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:07.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:07.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:08.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:07.723+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:08.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:08.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:08.135+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:08.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:08.135+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:08.872+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:08.872+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:08.872+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:08.872+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:08.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:08.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:08.679+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:08.679+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:09.184+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:09.184+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:09.184+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:09.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:09.184+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:09.848+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:09.848+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:09.848+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:10.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:09.848+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:09.645+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:09.645+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:10.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:09.645+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:10.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:09.645+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:10.194+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:10.194+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:10.194+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:10.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:10.194+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:10.850+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:10.850+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:10.850+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:10.850+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:10.634+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:10.634+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:10.634+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:11.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:10.634+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:11.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:11.224+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:11.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:11.224+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:11.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:11.224+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:11.521 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:11.224+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:11.825 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:11.680+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:11.825 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:11.680+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:11.826 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:11.680+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:11.826 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:11.680+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:12.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:11.825+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:12.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:11.825+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:12.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:11.825+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:12.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:11.825+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:12.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:12.149+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22671 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:12.181+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:12.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:12.181+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:12.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:12.181+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:12.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:12.181+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:12.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:12.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:12.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:12.819+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:13.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:12.651+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:13.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:12.651+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:13.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:12.651+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:13.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:12.651+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:13.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:13.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:13.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:13.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:13.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:13.199+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:13.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:13.199+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:13.823+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:13.823+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:13.823+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:14.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:13.823+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:13.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:13.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:13.643+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:14.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:13.643+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:14.593 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:14.211+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:14.593 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:14.211+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:14.593 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:14.211+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:14.593 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:14.211+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:14.859 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:14.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:14.859 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:14.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:14.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:14.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:14.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:14.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:15.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:14.859+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:14.859+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:14.859+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:14.859+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:15.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:15.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:15.163+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:15.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:15.163+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:15.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:15.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:15.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:16.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:15.888+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:15.698+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:15.698+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:15.698+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:16.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:15.698+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:16.620 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:16.200+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:16.200+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:16.200+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:16.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:16.200+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:17.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:16.906+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:17.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:16.906+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:17.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:16.906+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:17.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:16.906+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:17.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:16.673+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:16.673+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:16.673+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:16.673+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:17.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:17.153+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22765 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:17.203+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:17.203+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:17.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:17.203+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:17.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:17.203+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:17.866+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:17.866+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:17.866+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:18.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:17.866+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:17.639+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:17.639+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:17.639+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:18.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:17.639+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:18.221+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:18.221+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:18.221+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:18.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:18.221+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:18.831+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:18.831+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:18.831+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:18.832+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:18.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:18.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:18.642+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:18.642+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:19.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:19.249+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:19.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:19.249+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:19.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:19.249+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:19.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:19.249+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:19.858+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:19.858+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:19.858+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:19.858+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:19.626+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:19.626+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:19.626+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:19.626+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:20.273+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:20.273+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:20.273+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:20.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:20.273+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:20.836+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:20.836+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:20.836+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:20.836+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:20.670+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:20.670+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:20.670+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:20.670+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:21.319+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:21.319+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:21.319+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:21.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:21.319+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:21.855+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:21.855+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:21.855+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:21.855+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:22.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:21.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:21.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:21.712+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:22.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:21.712+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:22.155+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22863 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:22.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:22.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:22.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:22.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:22.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:22.308+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:22.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:22.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:22.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:22.819+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:22.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:22.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:22.692+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:22.692+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:23.624 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:23.278+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:23.685 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:23.278+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:23.685 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:23.278+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:23.686 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:23.278+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:23.783+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:23.783+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:23.783+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:23.783+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:23.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:23.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:23.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:24.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:23.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:24.620 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:24.300+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:24.300+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:24.300+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:24.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:24.300+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:24.764+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:24.764+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:24.764+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:25.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:24.764+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:24.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:24.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:24.689+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:25.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:24.689+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:25.299+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:25.299+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:25.299+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:25.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:25.299+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:25.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:25.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:25.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:25.729+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:25.805+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:25.805+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:25.805+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:25.805+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:26.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:26.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:26.336+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:26.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:26.336+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:26.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:26.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:26.730+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:26.730+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:26.853+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:26.853+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:26.853+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:26.853+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:27.160+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 22957 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:27.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:27.323+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:27.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:27.323+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:27.946 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:27.323+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:27.946 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:27.323+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:28.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:27.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:27.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:27.757+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:27.757+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:27.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:27.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:27.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:28.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:27.820+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:28.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:28.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:28.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:28.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:28.307+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:28.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:29.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:28.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:28.754+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:28.754+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:28.798+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:29.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:28.798+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:29.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:28.798+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:29.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:28.798+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:29.285+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:29.285+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:29.285+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:29.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:29.285+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:29.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:29.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:29.723+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:30.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:29.724+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:29.804+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:29.804+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:29.804+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:30.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:29.804+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:30.313+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:30.313+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:30.313+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:30.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:30.313+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:30.789+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:30.789+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:30.789+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:30.789+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:30.710+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:30.710+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:30.710+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:30.710+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:31.332+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:31.332+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:31.332+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:31.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:31.333+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:31.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:31.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:31.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:32.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:31.753+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:31.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:31.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:31.743+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:32.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:31.743+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:32.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:32.163+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23053 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:32.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:32.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:32.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:32.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:32.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:32.307+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:32.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:32.307+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:33.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:32.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:33.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:32.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:33.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:32.786+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:33.025 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:32.786+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:33.026 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:32.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:33.026 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:32.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:33.026 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:32.730+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:33.026 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:32.730+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:33.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:33.331+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:33.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:33.331+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:33.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:33.331+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:33.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:33.331+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:33.737+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:33.737+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:33.737+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:34.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:33.737+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:33.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:33.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:33.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:33.756+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:34.366+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:34.367+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:34.367+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:34.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:34.367+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:34.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:34.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:34.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:35.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:34.752+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:35.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:34.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:34.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:34.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:35.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:34.780+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:35.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:35.386+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:35.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:35.386+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:35.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:35.386+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:35.755 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:35.386+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:35.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:35.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:35.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:36.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:35.754+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:36.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:35.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:35.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:35.766+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:35.766+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:36.348+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:36.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:36.348+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:36.348+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:36.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:36.348+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:36.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:36.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:36.740+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:37.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:36.740+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:36.773+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:36.773+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:36.773+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:36.774+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:37.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:37.167+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23146 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:37.302+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:37.302+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:37.302+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:37.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:37.302+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:37.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:37.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:37.752+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:38.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:37.752+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:37.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:37.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:37.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:38.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:37.732+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:38.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:38.271+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:38.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:38.271+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:38.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:38.271+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:38.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:38.272+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:38.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:38.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:38.708+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:38.708+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:38.733+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:38.733+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:38.733+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:39.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:38.733+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:39.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:39.230+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:39.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:39.230+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:39.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:39.230+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:39.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:39.230+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:39.697+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:39.697+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:39.697+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:40.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:39.697+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:39.752+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:39.752+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:39.752+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:39.752+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:40.245+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:40.245+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:40.245+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:40.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:40.245+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:40.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:40.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:40.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:41.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:40.655+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:40.725+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:40.725+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:40.725+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:40.725+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:41.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:41.251+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:41.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:41.251+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:41.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:41.251+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:41.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:41.251+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:41.691+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:41.691+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:41.691+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:41.691+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:41.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:41.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:41.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:41.693+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:42.620 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:42.170+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23244 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:42.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:42.291+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:42.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:42.291+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:42.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:42.291+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:42.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:42.291+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:42.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:42.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:42.681+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:42.681+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:42.648+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:42.648+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:42.648+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:42.648+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:43.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:43.335+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:43.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:43.335+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:43.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:43.335+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:43.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:43.335+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:43.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:43.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:43.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:43.714+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:43.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:43.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:43.696+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:43.696+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:44.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:44.326+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:44.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:44.326+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:44.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:44.326+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:44.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:44.326+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:44.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:44.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:44.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:44.695+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:44.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:44.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:44.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:44.681+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:45.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:45.320+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:45.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:45.320+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:45.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:45.320+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:45.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:45.320+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:45.745+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:45.745+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:45.745+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:45.745+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:45.726+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:45.726+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:45.726+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:45.726+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:46.620 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:46.318+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:46.318+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:46.318+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:46.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:46.318+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:47.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:46.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:47.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:46.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:47.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:46.754+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:47.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:46.755+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:46.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:46.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:46.697+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:46.697+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:47.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:47.174+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23338 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:47.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:47.281+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:47.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:47.281+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:47.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:47.281+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:47.622 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:47.281+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:47.800+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:47.800+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:47.800+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:48.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:47.801+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:47.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:47.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:47.736+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:47.736+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:48.288+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:48.288+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:48.288+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:48.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:48.288+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:48.790+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:48.790+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:48.790+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:49.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:48.790+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:48.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:48.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:48.764+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:48.764+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:49.255+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:49.255+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:49.255+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:49.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:49.255+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:49.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:49.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:49.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:49.777+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:49.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:49.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:49.732+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:49.732+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:50.292+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:50.292+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:50.292+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:50.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:50.292+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:51.124 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:50.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:51.124 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:50.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:51.124 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:50.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:51.124 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:50.777+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:51.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:50.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:51.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:50.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:51.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:50.751+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:51.125 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:50.751+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:51.318+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:51.318+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:51.318+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:51.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:51.318+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:51.788+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:51.788+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:51.788+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:51.788+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:52.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:51.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:52.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:51.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:52.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:51.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:52.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:51.756+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:52.178+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23434 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:52.334+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:52.334+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:52.334+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:52.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:52.334+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:52.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:52.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:52.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:52.777+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:52.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:52.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:52.729+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:52.729+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:53.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:53.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:53.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:53.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:53.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:53.384+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:53.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:53.384+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:53.770+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:53.770+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:54.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:53.770+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:54.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:53.771+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:54.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:53.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:54.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:53.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:54.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:53.684+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:54.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:53.684+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:54.681 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:54.398+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:54.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:54.398+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:54.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:54.398+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:54.682 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:54.398+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:54.812+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:54.812+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:54.812+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:54.812+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:54.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:55.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:54.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:55.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:54.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:55.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:54.681+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:55.771 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:55.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:55.771 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:55.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:55.771 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:55.441+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:55.771 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:55.441+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:55.772 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:55.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:55.772 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:55.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:55.772 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:55.647+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:55.772 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:55.647+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:55.770+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:55.770+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:55.770+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:56.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:55.770+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:56.776 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:56.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:56.776 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:56.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:56.776 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:56.430+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:56.776 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:56.430+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:56.777 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:56.660+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:56.777 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:56.660+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:56.777 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:56.660+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:56.777 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:56.660+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:57.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:56.775+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:57.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:56.775+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:57.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:56.775+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:57.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:56.775+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:57.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:07:57.181+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23528 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:07:57.820 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:57.466+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:57.820 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:57.466+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:57.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:57.466+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:57.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:57.466+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:57.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:57.645+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:57.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:57.645+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:57.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:57.645+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:57.821 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:57.645+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:58.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:57.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:58.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:57.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:58.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:57.820+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:58.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:57.820+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:58.799 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:58.436+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:58.799 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:58.436+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:58.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:58.436+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:58.800 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:58.437+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:58.800 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:58.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:58.800 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:58.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:58.800 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:58.650+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:58.801 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:58.650+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:59.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:58.798+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:58.798+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:58.798+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:07:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:58.798+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:59.834 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:59.480+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:59.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:59.480+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:59.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:59.480+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:07:59.835 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:07:59.480+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:07:59.835 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:59.622+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:59.835 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:59.622+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:59.835 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:59.622+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:07:59.836 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:07:59.622+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:59.833+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:59.833+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:59.833+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:00.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:07:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:07:59.833+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:00.829 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:00.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:00.829 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:00.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:00.830 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:00.495+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:00.830 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:00.495+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:00.830 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:00.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:00.830 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:00.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:00.830 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:00.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:00.830 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:00.610+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:00.828+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:00.828+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:00.828+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:01.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:00.828+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:01.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:01.541+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:01.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:01.541+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:01.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:01.541+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:01.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:01.541+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:01.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:01.577+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:01.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:01.577+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:01.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:01.577+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:01.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:01.577+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:02.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:01.859+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:02.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:01.859+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:02.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:01.859+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:02.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:01.859+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:02.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:02.184+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23625 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:02.555+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:02.555+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:02.555+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:02.555+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:02.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:02.616+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:02.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:02.616+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:02.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:02.616+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:02.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:02.616+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:02.886+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:02.886+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:02.886+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:03.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:02.887+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:03.745 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:03.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:03.746 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:03.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:03.746 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:03.519+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:03.746 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:03.519+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:03.746 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:03.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:03.746 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:03.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:03.747 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:03.610+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:03.747 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:03.610+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:03.935+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:03.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:03.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:03.936+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:04.557+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:04.557+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:04.557+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:04.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:04.557+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:04.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:04.582+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:04.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:04.582+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:04.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:04.582+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:04.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:04.582+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:04.911+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:04.911+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:04.911+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:05.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:04.911+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:05.510+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:05.510+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:05.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:05.510+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:05.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:05.510+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:05.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:05.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:05.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:05.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:05.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:05.619+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:05.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:05.619+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:05.892+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:05.892+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:05.892+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:06.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:05.892+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:06.537+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:06.537+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:06.537+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:06.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:06.537+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:06.639+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:06.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:06.639+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:06.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:06.639+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:06.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:06.639+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:07.189 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:06.882+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:07.190 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:06.882+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:07.190 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:06.882+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:07.190 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:06.882+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:07.542 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:07.189+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23720 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:07.541+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:07.541+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:07.541+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:07.541+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:07.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:07.631+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:07.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:07.631+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:07.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:07.631+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:07.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:07.631+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:07.904+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:07.904+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:07.904+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:08.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:07.904+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:08.574+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:08.574+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:08.574+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:08.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:08.574+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:08.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:08.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:08.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:08.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:08.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:08.632+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:08.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:08.632+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:08.920+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:08.920+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:08.920+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:08.921+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:09.538+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:09.538+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:09.538+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:09.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:09.539+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:09.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:09.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:09.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:09.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:09.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:09.613+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:09.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:09.613+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:10.191 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:09.906+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:10.191 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:09.906+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:10.191 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:09.906+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:10.191 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:09.906+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:10.869 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:10.631+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:10.631+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:10.631+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:10.631+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:10.543+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:10.543+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:10.543+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:10.870 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:10.543+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:10.869+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:10.869+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:10.869+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:11.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:10.869+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:11.614+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:11.614+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:11.614+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:11.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:11.615+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:11.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:11.583+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:11.583+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:11.583+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:11.583+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:12.193 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:11.899+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:12.194 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:11.899+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:12.194 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:11.899+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:12.194 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:11.899+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:12.567 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:12.192+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23816 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:12.854 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:12.593+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:12.855 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:12.593+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:12.855 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:12.593+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:12.855 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:12.593+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:12.855 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:12.567+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:12.855 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:12.567+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:12.855 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:12.567+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:12.856 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:12.567+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:12.853+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:12.853+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:12.853+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:13.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:12.854+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:13.615+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:13.615+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:13.615+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:13.616+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:13.809+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:13.809+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:13.809+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:13.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:13.809+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:13.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:13.592+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:13.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:13.592+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:13.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:13.592+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:13.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:13.592+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:14.578+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:14.578+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:14.578+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:14.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:14.578+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:14.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:14.628+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:14.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:14.628+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:14.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:14.628+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:14.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:14.628+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:14.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:14.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:14.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:14.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:14.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:14.796+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:14.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:14.796+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:15.607+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:15.607+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:15.607+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:15.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:15.607+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:15.621+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:15.621+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:15.621+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:15.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:15.621+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:15.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:15.814+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:15.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:15.814+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:15.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:15.814+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:15.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:15.814+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:16.620+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:16.620+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:16.620+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:17.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:16.621+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:16.661+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:16.661+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:16.661+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:17.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:16.661+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:17.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:16.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:17.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:16.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:17.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:16.781+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:17.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:16.781+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:17.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:17.196+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 23911 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:17.668+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:17.668+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:17.668+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:18.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:17.668+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:17.826+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:17.826+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:17.826+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:17.826+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:17.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:17.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:18.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:17.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:18.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:17.640+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:18.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:18.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:18.663+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:19.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:18.663+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:19.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:18.844+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:19.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:18.844+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:19.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:18.844+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:19.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:18.844+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:18.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:19.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:18.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:19.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:18.645+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:19.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:18.645+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:19.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:19.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:19.644+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:20.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:19.644+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:19.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:19.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:19.819+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:19.819+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:19.628+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:19.628+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:19.628+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:20.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:19.628+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:20.860 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:20.588+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:20.861 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:20.588+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:20.861 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:20.588+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:20.861 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:20.588+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:20.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:20.596+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:20.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:20.596+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:20.861 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:20.596+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:20.862 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:20.596+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:20.860+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:20.860+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:20.860+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:20.860+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:21.857 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:21.615+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:21.857 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:21.615+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:21.857 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:21.615+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:21.858 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:21.615+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:21.858 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:21.601+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:21.858 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:21.601+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:21.858 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:21.601+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:21.858 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:21.601+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:21.856+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:21.856+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:21.856+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:21.856+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:22.555 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:22.200+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24008 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:22.623+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:22.623+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:22.623+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:22.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:22.623+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:22.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:22.555+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:22.555+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:22.555+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:22.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:22.555+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:22.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:22.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:22.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:22.888+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:23.657+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:23.657+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:23.657+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:23.657+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:23.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:23.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:23.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:23.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:23.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:23.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:23.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:23.600+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:23.901+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:23.901+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:23.901+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:24.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:23.901+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:24.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:24.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:24.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:24.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:24.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:24.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:24.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:24.652+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:24.553+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:24.553+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:24.553+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:24.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:24.553+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:24.874+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:24.874+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:24.874+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:25.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:24.874+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:25.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:25.680+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:25.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:25.680+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:25.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:25.680+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:25.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:25.680+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:25.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:25.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:25.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:25.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:25.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:25.600+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:25.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:25.600+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:26.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:25.923+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:25.923+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:25.923+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:26.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:25.923+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:26.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:26.688+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:26.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:26.688+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:26.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:26.688+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:26.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:26.688+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:26.612+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:26.612+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:26.612+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:26.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:26.612+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:27.205 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:26.885+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:27.205 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:26.885+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:27.206 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:26.885+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:27.206 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:26.885+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:27.204+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24103 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:27.936 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:27.652+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:27.936 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:27.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:27.936 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:27.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:27.936 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:27.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:27.937 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:27.647+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:27.937 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:27.647+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:27.937 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:27.647+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:27.937 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:27.647+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:28.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:27.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:27.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:27.936+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:27.936+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:28.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:28.665+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:28.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:28.665+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:28.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:28.665+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:28.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:28.665+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:28.971 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:28.689+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:28.971 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:28.689+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:28.971 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:28.689+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:28.971 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:28.689+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:28.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:28.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:28.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:28.970+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:29.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:29.694+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:29.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:29.694+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:29.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:29.694+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:29.968 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:29.694+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:29.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:29.660+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:29.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:29.660+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:29.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:29.660+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:29.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:29.660+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:29.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:29.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:29.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:30.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:29.967+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:31.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:30.693+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:31.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:30.693+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:31.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:30.693+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:31.016 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:30.693+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:31.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:30.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:31.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:30.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:31.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:30.693+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:31.016 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:30.693+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:31.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:31.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:31.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:31.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:31.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:31.015+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:32.055 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:31.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:32.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:31.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:32.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:31.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:32.056 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:31.690+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:32.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:31.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:32.056 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:31.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:32.057 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:31.653+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:32.057 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:31.653+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:32.336 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:32.208+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24198 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:32.336 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:32.054+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:32.336 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:32.054+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:32.336 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:32.054+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:32.337 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:32.055+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:33.104 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:32.715+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:33.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:32.715+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:33.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:32.715+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:33.105 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:32.715+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:33.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:32.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:33.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:32.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:33.105 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:32.688+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:33.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:32.688+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:33.104+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:33.104+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:33.104+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:33.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:33.104+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:34.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:33.714+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:34.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:33.714+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:34.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:33.714+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:34.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:33.715+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:34.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:33.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:34.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:33.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:34.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:33.681+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:34.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:33.681+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:34.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:34.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:34.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:34.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:34.079+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:34.723 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:34.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:34.723 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:34.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:34.723 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:34.687+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:34.723 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:34.687+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:35.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:34.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:35.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:34.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:35.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:34.722+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:35.067 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:34.722+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:35.067+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:35.067+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:35.067+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:35.067+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:36.059 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:35.644+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:36.059 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:35.644+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:36.059 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:35.644+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:36.060 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:35.644+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:36.060 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:35.753+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:36.060 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:35.753+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:36.060 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:35.753+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:36.060 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:35.753+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:36.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:36.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:36.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:36.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:36.057+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:36.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:36.604+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:36.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:36.604+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:36.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:36.604+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:36.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:36.604+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:36.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:36.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:36.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:36.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:36.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:36.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:36.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:36.802+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:37.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:37.212+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24293 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:37.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:37.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:37.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:37.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:37.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:37.063+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:37.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:37.063+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:38.097 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:37.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:38.098 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:37.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:38.098 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:37.799+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:38.098 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:37.799+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:38.098 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:37.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:38.098 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:37.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:38.099 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:37.640+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:38.099 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:37.640+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:38.097+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:38.097+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:38.097+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:38.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:38.097+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:39.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:38.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:39.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:38.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:39.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:38.815+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:39.124 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:38.815+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:39.124 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:38.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:39.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:38.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:39.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:38.652+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:39.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:38.652+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:39.627 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:39.137+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:39.627 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:39.137+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:39.627 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:39.137+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:39.628 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:39.137+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:39.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:39.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:39.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:39.828+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:39.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:39.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:39.651+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:39.651+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:40.608 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:40.134+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:40.609 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:40.134+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:40.609 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:40.134+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:40.609 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:40.135+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:40.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:40.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:40.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:40.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:40.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:40.655+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:40.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:40.655+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:41.166 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:40.874+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:41.167 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:40.874+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:41.167 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:40.874+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:41.167 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:40.874+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:41.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:41.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:41.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:41.165+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:41.696+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:41.696+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:41.696+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:41.696+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:41.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:41.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:41.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:42.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:41.885+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:42.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:42.215+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24390 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:42.148+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:42.148+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:42.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:42.148+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:42.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:42.148+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:42.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:42.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:42.845+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:42.845+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:42.654+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:42.654+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:42.654+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:42.654+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:43.120+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:43.120+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:43.120+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:43.121+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:43.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:43.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:43.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:43.888+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:43.700+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:43.700+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:43.700+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:43.700+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:44.128+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:44.128+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:44.128+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:44.128+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:44.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:44.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:45.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:44.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:44.929+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:44.704+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:44.704+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:44.704+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:44.704+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:45.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:45.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:45.165+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:45.165+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:45.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:45.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:45.912+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:46.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:45.912+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:45.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:45.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:45.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:45.735+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:46.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:46.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:46.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:46.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:46.197+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:46.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:46.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:47.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:46.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:46.937+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:46.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:46.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:46.779+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:47.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:46.780+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:47.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:47.220+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24485 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:47.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:47.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:47.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:47.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:47.149+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:47.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:47.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:47.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:48.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:47.925+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:47.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:47.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:47.774+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:48.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:47.775+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:48.196+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:48.196+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:48.196+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:48.196+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:48.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:48.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:48.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:49.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:48.888+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:48.775+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:48.775+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:48.775+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:48.775+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:49.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:49.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:49.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:49.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:49.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:49.210+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:49.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:49.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:49.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:50.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:49.931+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:49.800+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:49.800+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:49.800+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:49.801+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:50.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:50.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:50.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:50.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:50.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:50.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:50.173+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:50.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:50.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:51.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:50.974+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:50.974+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:50.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:50.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:50.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:51.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:50.776+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:51.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:51.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:51.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:51.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:51.147+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:52.107 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:51.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:52.107 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:51.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:52.107 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:51.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:52.107 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:51.799+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:52.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:51.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:52.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:51.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:52.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:51.966+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:52.108 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:51.966+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:52.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:52.223+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24609 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:52.106+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:52.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:52.106+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:52.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:52.106+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:52.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:52.106+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:52.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:52.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:53.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:52.957+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:52.957+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:52.764+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:52.764+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:52.764+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:52.764+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:53.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:53.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:53.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:53.155+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:53.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:53.155+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:53.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:53.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:53.936+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:53.936+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:53.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:53.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:53.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:53.804+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:54.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:54.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:54.173+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:54.173+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:54.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:54.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:54.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:54.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:54.777+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:54.777+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:54.777+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:54.777+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:55.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:55.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:55.207+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:55.207+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:55.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:55.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:55.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:55.902+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:55.759+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:55.759+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:55.759+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:55.759+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:56.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:56.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:56.253+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:56.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:56.253+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:56.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:56.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:56.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:56.950+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:56.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:56.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:56.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:56.758+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:57.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:08:57.227+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24703 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:08:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:57.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:57.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:57.275+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:57.276+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:57.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:57.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:57.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:57.773+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:58.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:57.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:58.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:57.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:58.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:57.915+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:58.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:57.915+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:58.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:58.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:58.270+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:58.270+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:58.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:58.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:58.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:08:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:58.776+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:58.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:58.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:58.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:08:59.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:58.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:08:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:59.241+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:59.241+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:59.241+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:08:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:08:59.241+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:59.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:59.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:59.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:08:59.778+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:59.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:59.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:59.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:08:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:08:59.944+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:00.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:00.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:00.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:00.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:00.197+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:00.738+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:00.738+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:00.738+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:01.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:00.739+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:00.963+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:00.963+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:00.963+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:00.963+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:01.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:01.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:01.205+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:01.205+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:02.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:01.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:02.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:01.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:02.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:01.919+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:02.113 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:01.919+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:02.114 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:01.692+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:02.114 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:01.692+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:02.114 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:01.692+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:02.114 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:01.692+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:02.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:02.231+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24800 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:02.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:02.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:02.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:02.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:02.170+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:02.691+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:02.691+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:02.691+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:02.691+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:02.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:02.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:02.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:02.906+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:03.212+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:03.212+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:03.212+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:03.212+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:03.737+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:03.737+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:03.737+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:03.737+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:03.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:03.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:03.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:04.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:03.861+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:04.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:04.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:04.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:04.184+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:04.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:04.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:04.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:04.778+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:04.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:04.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:04.888+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:04.888+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:05.623 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:05.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:05.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:05.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:05.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:05.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:05.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:05.201+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:05.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:05.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:05.804+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:05.804+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:05.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:05.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:05.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:06.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:05.848+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:06.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:06.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:06.158+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:06.158+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:06.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:06.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:07.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:06.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:06.838+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:06.853+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:06.853+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:06.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:07.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:06.854+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:07.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:07.235+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24894 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:07.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:07.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:07.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:07.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:07.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:07.170+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:07.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:07.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:07.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:07.869+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:07.807+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:07.807+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:07.807+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:07.807+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:08.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:08.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:08.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:08.197+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:08.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:08.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:08.840+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:09.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:08.840+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:08.767+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:08.767+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:08.767+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:09.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:08.768+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:09.204+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:09.204+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:09.204+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:09.204+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:09.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:09.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:09.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:09.828+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:09.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:09.736+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:09.736+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:09.736+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:10.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:10.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:10.174+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:10.174+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:10.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:10.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:10.792+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:10.792+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:10.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:10.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:10.755+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:10.755+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:11.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:11.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:11.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:11.140+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:11.781+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:11.781+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:11.781+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:11.781+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:11.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:11.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:11.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:11.726+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:12.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:12.239+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 24989 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:12.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:12.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:12.130+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:12.130+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:12.778+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:12.778+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:12.778+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:13.120 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:12.778+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:12.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:12.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:12.735+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:13.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:12.735+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:13.120+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:13.120+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:13.120+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:13.120+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:14.085 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:13.793+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:14.086 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:13.793+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:14.086 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:13.793+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:14.086 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:13.793+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:14.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:13.731+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:14.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:13.731+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:14.087 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:13.731+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:14.087 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:13.731+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:14.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:14.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:14.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:14.085+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:15.068 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:14.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:15.068 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:14.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:15.068 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:14.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:15.068 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:14.716+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:15.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:14.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:15.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:14.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:15.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:14.842+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:15.069 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:14.842+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:15.066+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:15.066+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:15.066+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:15.066+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:16.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:15.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:16.047 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:15.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:16.048 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:15.829+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:16.048 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:15.829+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:16.048 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:15.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:16.048 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:15.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:16.048 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:15.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:16.048 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:15.706+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:16.047+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:16.047+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:16.047+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:16.047+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:17.079 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:16.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:17.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:16.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:17.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:16.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:17.080 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:16.720+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:17.080 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:16.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:17.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:16.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:17.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:16.820+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:17.081 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:16.820+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:17.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:17.242+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25083 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:17.370 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:17.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:17.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:17.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:17.078+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:18.091 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:17.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:18.091 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:17.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:18.091 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:17.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:18.091 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:17.701+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:18.091 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:17.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:18.092 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:17.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:18.092 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:17.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:18.092 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:17.868+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:18.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:18.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:18.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:18.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:18.089+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:19.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:18.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:19.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:18.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:19.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:18.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:19.071 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:18.750+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:19.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:18.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:19.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:18.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:19.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:18.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:19.072 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:18.900+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:19.069+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:19.069+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:19.069+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:19.069+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:20.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:19.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:20.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:19.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:20.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:19.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:20.083 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:19.730+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:20.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:19.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:20.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:19.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:20.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:19.869+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:20.084 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:19.869+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:20.082+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:20.082+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:20.082+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:20.082+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:20.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:20.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:20.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:20.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:20.706+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:21.125+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:21.125+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:21.125+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:21.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:21.125+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:21.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:20.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:20.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:20.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:20.900+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:22.109 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:21.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:22.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:21.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:22.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:21.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:22.110 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:21.879+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:22.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:21.676+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:22.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:21.676+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:22.110 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:21.676+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:22.111 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:21.676+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:22.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:22.246+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25179 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:22.107+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:22.107+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:22.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:22.107+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:22.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:22.108+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:23.116 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:22.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:23.116 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:22.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:23.116 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:22.662+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:23.117 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:22.662+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:23.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:22.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:23.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:22.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:23.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:22.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:23.117 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:22.870+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:23.115+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:23.115+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:23.115+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:23.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:23.115+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:23.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:23.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:23.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:23.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:23.658+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:23.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:23.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:23.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:23.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:23.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:23.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:23.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:23.854+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:24.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:24.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:24.146+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:24.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:24.147+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:24.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:24.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:24.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:24.707+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:24.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:24.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:24.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:25.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:24.868+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:25.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:25.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:25.149+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:25.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:25.149+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:25.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:25.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:25.711+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:25.711+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:25.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:25.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:25.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:26.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:25.854+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:26.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:26.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:26.170+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:26.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:26.171+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:26.666+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:26.666+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:26.666+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:26.666+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:26.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:26.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:26.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:27.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:26.881+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:27.251+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25273 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:27.200+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:27.200+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:27.200+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:27.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:27.200+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:27.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:27.634+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:27.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:27.634+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:27.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:27.634+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:27.912 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:27.634+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:28.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:28.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:28.154+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:28.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:28.154+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:28.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:27.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:27.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:27.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:27.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:28.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:28.602+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:28.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:28.602+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:28.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:28.602+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:28.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:28.602+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:29.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:29.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:29.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:29.140+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:29.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:29.140+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:28.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:28.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:28.951+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:28.952+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:29.617+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:29.617+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:29.617+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:29.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:29.618+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:30.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:29.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:30.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:29.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:30.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:29.917+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:30.187 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:29.917+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:30.185+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:30.185+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:30.185+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:30.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:30.186+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:30.913 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:30.637+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:30.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:30.637+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:30.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:30.637+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:30.914 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:30.637+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:31.177 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:30.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:31.178 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:30.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:31.178 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:30.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:31.178 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:30.913+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:31.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:31.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:31.177+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:31.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:31.177+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:31.947 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:31.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:31.947 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:31.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:31.947 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:31.641+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:31.948 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:31.641+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:32.256 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:32.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:32.256 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:32.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:32.256 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:32.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:32.257 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:32.185+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:32.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:31.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:32.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:31.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:32.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:31.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:32.257 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:31.946+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:32.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:32.255+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25368 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:32.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:32.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:32.658+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:32.658+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:32.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:32.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:32.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:32.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:33.188+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:33.188+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:33.188+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:33.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:33.188+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:33.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:33.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:33.690+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:33.690+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:33.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:33.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:33.885+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:33.885+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:34.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:34.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:34.184+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:34.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:34.184+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:34.682+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:34.682+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:34.682+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:34.682+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:34.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:34.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:34.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:34.927+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:35.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:35.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:35.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:35.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:35.203+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:35.689+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:35.689+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:35.689+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:36.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:35.689+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:35.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:35.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:35.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:35.903+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:36.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:36.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:36.227+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:36.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:36.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:36.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:36.686+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:37.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:36.686+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:36.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:36.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:36.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:37.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:36.856+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:37.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:37.259+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25463 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:37.200+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:37.200+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:37.200+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:37.200+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:37.889+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:37.889+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:37.889+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:38.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:37.889+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:37.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:37.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:37.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:38.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:37.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:38.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:38.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:38.197+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:38.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:38.197+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:38.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:38.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:38.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:38.856+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:38.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:38.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:38.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:38.712+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:39.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:39.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:39.206+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:39.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:39.206+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:39.817+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:39.817+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:40.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:39.817+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:40.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:39.817+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:39.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:39.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:39.702+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:40.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:39.702+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:40.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:40.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:40.159+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:40.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:40.159+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:40.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:40.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:40.854+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:41.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:40.854+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:40.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:40.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:40.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:40.728+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:41.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:41.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:41.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:41.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:41.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:41.203+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:41.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:41.204+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:42.049 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:41.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:42.049 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:41.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:42.049 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:41.776+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:42.050 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:41.776+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:42.050 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:41.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:42.050 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:41.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:42.050 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:41.887+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:42.051 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:41.887+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:42.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:42.263+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25560 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:42.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:42.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:42.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:42.191+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:42.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:42.191+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:42.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:42.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:42.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:42.758+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:42.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:42.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:42.868+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:43.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:42.868+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:43.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:43.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:43.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:43.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:43.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:43.210+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:43.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:43.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:43.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:43.797+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:43.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:43.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:43.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:43.846+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:44.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:44.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:44.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:44.201+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:44.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:44.201+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:44.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:44.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:44.797+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:45.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:44.797+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:44.821+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:44.821+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:44.821+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:45.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:44.822+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:45.186+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:45.186+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:45.186+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:45.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:45.186+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:45.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:45.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:45.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:46.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:45.784+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:45.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:45.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:45.782+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:46.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:45.782+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:46.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:46.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:46.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:46.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:46.210+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:46.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:46.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:46.818+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:46.819+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:46.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:46.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:46.777+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:46.777+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:47.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:47.267+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25652 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:47.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:47.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:47.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:47.246+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:47.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:47.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:47.852+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:47.852+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:47.763+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:47.763+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:47.763+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:48.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:47.763+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:48.296+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:48.296+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:48.296+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:48.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:48.296+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:48.801+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:48.801+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:48.801+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:49.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:48.801+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:49.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:48.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:48.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:48.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:48.881+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:49.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:49.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:49.274+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:49.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:49.275+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:49.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:49.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:49.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:49.916+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:49.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:49.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:49.756+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:50.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:49.756+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:50.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:50.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:50.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:50.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:50.322+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:50.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:50.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:50.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:51.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:50.874+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:50.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:50.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:50.784+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:51.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:50.784+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:51.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:51.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:51.305+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:51.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:51.305+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:52.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:51.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:51.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:51.827+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:51.827+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:52.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:51.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:52.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:51.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:52.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:51.780+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:52.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:51.780+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:52.271+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25748 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:52.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:52.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:52.327+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:52.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:52.328+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:52.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:52.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:52.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:52.830+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:52.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:52.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:52.783+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:53.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:52.783+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:53.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:53.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:53.320+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:53.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:53.320+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:53.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:53.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:53.795+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:54.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:53.795+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:53.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:53.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:53.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:53.817+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:54.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:54.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:54.277+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:54.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:54.277+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:54.827+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:54.827+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:54.827+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:55.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:54.827+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:54.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:54.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:54.855+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:54.855+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:55.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:55.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:55.285+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:55.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:55.285+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:55.827+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:55.827+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:55.827+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:56.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:55.827+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:56.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:55.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:55.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:55.864+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:55.864+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:56.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:56.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:56.322+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:56.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:56.322+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:56.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:56.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:56.816+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:57.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:56.816+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:56.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:56.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:56.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:57.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:56.901+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:57.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:09:57.273+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25843 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:09:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:57.287+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:57.287+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:57.287+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:57.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:57.287+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:57.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:57.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:57.819+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:58.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:57.819+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:57.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:57.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:57.881+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:58.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:57.881+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:58.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:58.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:58.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:58.334+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:58.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:58.334+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:58.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:58.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:58.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:09:59.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:58.846+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:58.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:58.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:58.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:09:59.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:58.901+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:09:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:59.328+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:59.328+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:59.328+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:09:59.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:09:59.328+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:59.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:59.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:59.861+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:09:59.861+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:59.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:59.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:59.839+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:00.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:09:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:09:59.839+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:00.480 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:00.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:00.481 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:00.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:00.481 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:00.309+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:00.481 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:00.309+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:00.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:00.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:00.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:00.890+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:01.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:00.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:00.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:00.848+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:01.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:00.848+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:01.302+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:01.302+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:01.302+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:01.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:01.302+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:01.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:01.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:01.867+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:01.867+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:02.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:01.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:02.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:01.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:02.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:01.876+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:02.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:01.876+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:02.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:02.278+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 25940 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:02.293+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:02.293+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:02.293+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:02.293+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:02.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:02.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:02.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:03.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:02.841+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:02.889+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:02.889+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:02.889+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:03.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:02.889+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:03.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:03.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:03.245+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:03.553 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:03.245+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:03.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:03.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:03.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:03.859+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:03.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:03.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:03.805+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:03.805+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:04.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:04.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:04.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:04.210+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:04.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:04.211+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:04.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:04.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:05.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:04.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:05.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:04.903+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:04.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:04.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:04.788+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:04.788+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:05.199+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:05.199+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:05.199+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:05.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:05.199+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:05.858+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:05.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:05.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:06.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:05.859+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:05.742+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:05.742+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:05.742+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:05.742+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:06.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:06.226+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:06.226+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:06.226+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:06.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:06.227+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:06.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:06.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:06.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:07.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:06.904+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:06.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:06.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:06.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:07.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:06.743+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:07.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:07.282+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26035 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:07.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:07.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:07.193+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:07.193+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:07.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:07.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:07.859+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:08.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:07.859+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:07.785+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:07.785+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:07.785+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:07.785+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:08.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:08.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:08.181+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:08.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:08.181+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:08.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:08.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:08.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:09.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:08.909+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:08.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:08.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:08.817+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:08.817+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:09.142+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:09.142+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:09.142+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:09.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:09.142+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:09.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:09.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:09.880+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:10.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:09.880+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:09.787+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:09.787+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:09.787+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:10.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:09.787+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:10.139+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:10.139+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:10.139+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:10.139+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:10.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:10.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:10.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:11.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:10.924+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:10.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:10.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:10.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:11.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:10.758+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:11.127+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:11.127+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:11.127+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:11.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:11.128+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:11.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:11.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:11.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:11.959+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:11.731+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:11.731+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:11.731+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:12.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:11.731+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:12.410 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:12.286+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26130 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:12.411 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:12.132+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:12.411 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:12.132+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:12.411 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:12.132+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:12.411 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:12.132+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:13.086 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:12.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:13.087 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:12.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:13.087 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:12.756+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:13.087 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:12.756+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:13.088 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:12.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:13.088 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:12.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:13.088 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:12.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:13.089 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:12.993+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:13.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:13.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:13.085+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:13.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:13.085+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:14.023 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:13.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:14.024 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:13.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:14.024 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:13.710+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:14.024 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:13.710+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:14.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:14.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:14.089+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:14.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:14.089+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.023+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.023+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.023+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.023+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:14.991 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:14.674+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:14.992 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:14.674+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:14.992 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:14.674+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:14.992 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:14.674+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.991+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:15.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:14.991+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:15.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:15.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:15.079+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:15.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:15.079+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:15.978 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:15.665+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:15.979 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:15.665+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:15.979 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:15.665+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:15.979 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:15.665+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:15.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:15.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:15.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:15.978+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:16.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:16.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:16.078+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:16.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:16.078+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:16.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:16.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:16.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:16.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:16.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:16.701+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:16.977 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:16.701+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:17.291 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:17.034+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:17.292 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:17.034+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:17.292 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:17.034+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:17.292 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:17.034+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:17.292 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:16.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:17.292 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:16.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:17.292 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:16.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:17.293 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:16.976+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:17.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:17.291+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26225 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:18.007 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:17.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:18.007 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:17.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:18.007 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:17.739+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:18.007 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:17.739+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.006+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:18.046+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:18.046+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:18.046+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:18.046+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.000+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:19.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.000+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.975+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:19.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:18.975+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:18.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:19.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:18.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:19.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:18.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:19.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:18.769+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.987+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.987+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.987+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:20.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:19.987+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:19.790+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:19.790+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:19.790+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:19.790+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:19.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:19.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:20.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:19.947+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:20.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:19.947+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:20.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:20.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:20.967+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:20.968+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:20.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:20.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:20.903+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:21.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:20.903+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:21.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:20.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:21.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:20.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:21.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:20.825+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:21.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:20.825+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:21.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:21.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:21.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:21.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:21.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:21.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:21.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:21.801+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:21.986 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:21.960+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:21.986 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:21.960+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:21.986 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:21.960+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:21.986 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:21.960+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:21.986 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:21.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:21.987 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:21.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:21.987 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:21.872+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:21.987 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:21.872+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:22.295+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26322 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:22.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:22.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:23.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:22.847+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:23.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:22.847+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:22.922+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:22.922+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:22.922+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:23.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:22.923+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:22.796+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:22.796+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:22.796+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:23.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:22.796+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:23.857+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:23.857+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:23.857+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:24.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:23.857+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:23.915+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:23.915+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:23.915+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:23.915+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:24.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:23.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:24.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:23.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:24.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:23.799+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:24.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:23.799+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:24.895+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:24.895+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:24.895+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:25.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:24.895+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:25.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:24.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:25.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:24.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:25.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:24.927+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:25.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:24.927+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:25.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:24.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:24.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:24.750+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:25.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:24.750+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:25.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:25.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:25.942+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:26.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:25.942+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:25.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:25.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:25.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:25.888+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:26.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:25.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:25.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:25.769+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:26.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:25.769+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:26.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:26.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:26.906+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:27.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:26.906+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:26.839+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:26.839+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:26.839+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:26.839+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:26.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:26.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:27.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:26.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:27.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:26.727+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:27.299+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26417 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:27.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:27.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:27.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:28.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:27.712+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:27.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:27.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:27.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:27.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:27.810+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:27.810+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:28.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:27.810+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:28.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:27.810+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:28.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:28.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:28.774+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:28.774+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:28.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:28.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:28.758+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:29.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:28.758+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:28.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:29.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:28.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:29.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:28.879+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:29.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:28.879+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:29.763+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:29.763+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:29.763+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:30.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:29.763+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:29.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:29.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:29.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:29.778+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:29.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:29.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:30.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:29.899+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:30.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:29.899+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:30.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:30.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:30.784+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:31.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:30.784+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:30.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:30.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:30.865+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:31.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:30.865+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:31.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:30.812+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:31.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:30.812+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:31.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:30.812+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:31.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:30.812+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:31.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:31.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:31.905+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:32.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:31.905+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:32.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:31.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:32.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:31.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:32.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:31.778+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:32.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:31.778+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:32.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:31.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:32.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:31.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:32.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:31.777+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:32.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:31.778+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:32.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:32.303+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26512 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:32.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:32.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:32.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:32.792+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:33.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:32.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:32.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:32.870+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:32.870+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:32.735+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:32.735+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:32.735+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:33.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:32.736+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:33.826+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:33.826+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:33.826+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:34.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:33.826+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:33.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:33.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:33.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:34.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:33.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:34.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:33.688+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:34.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:33.688+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:34.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:33.688+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:34.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:33.688+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:34.789+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:34.789+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:35.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:34.789+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:35.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:34.789+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:35.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:34.654+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:35.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:34.654+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:35.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:34.654+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:35.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:34.654+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:34.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:34.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:34.900+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:35.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:34.900+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:35.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:35.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:35.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:36.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:35.821+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:36.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:35.644+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:36.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:35.644+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:36.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:35.644+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:36.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:35.644+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:35.855+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:36.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:35.855+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:35.855+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:36.123 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:35.855+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:36.850 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:36.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:36.850 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:36.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:36.850 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:36.803+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:36.850 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:36.803+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:36.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:36.595+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:36.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:36.595+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:36.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:36.595+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:36.851 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:36.595+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:36.849+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:36.849+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:36.849+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:37.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:36.849+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:37.600 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:37.306+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26606 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:37.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:37.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:37.802+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:37.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:37.802+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:37.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:37.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:37.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:37.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:37.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:37.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:37.599+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:37.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:37.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:37.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:37.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:37.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:37.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:37.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:37.832+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:38.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:38.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:38.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:38.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:38.848 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:38.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:38.849 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:38.562+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:38.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:38.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:38.846+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:39.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:38.846+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:38.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:38.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:38.872+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:38.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:39.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:39.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:39.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:39.572+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:40.200 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:39.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:40.200 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:39.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:40.200 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:39.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:40.200 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:39.890+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:40.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:39.894+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:40.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:39.894+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:40.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:39.894+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:40.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:39.894+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:40.861 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:40.609+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:40.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:40.609+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:40.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:40.609+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:40.862 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:40.609+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:40.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:40.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:40.892+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:40.892+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:40.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:40.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:40.860+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:41.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:40.860+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:41.604+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:41.604+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:41.604+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:41.604+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:42.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:41.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:42.316 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:41.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:42.316 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:41.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:42.316 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:41.907+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:42.316 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:41.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:42.317 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:41.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:42.317 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:41.881+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:42.317 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:41.881+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:42.562 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:42.311+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26703 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:42.563 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:42.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:42.563 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:42.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:42.563 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:42.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:42.563 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:42.562+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:42.866+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:42.866+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:42.866+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:43.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:42.866+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:43.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:42.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:42.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:42.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:43.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:42.882+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:43.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:43.548+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:43.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:43.548+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:43.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:43.548+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:43.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:43.548+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:43.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:43.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:43.846+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:44.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:43.846+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:44.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:43.838+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:43.838+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:43.838+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:44.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:43.838+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:44.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:44.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:44.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:44.572+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:44.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:44.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:44.885+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:45.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:44.885+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:44.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:44.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:44.877+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:45.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:44.877+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:45.609+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:45.609+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:45.609+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:45.609+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:45.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:45.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:45.828+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:45.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:45.828+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:45.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:45.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:45.887+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:45.887+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:46.856 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:46.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:46.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:46.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:46.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:46.587+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:46.857 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:46.587+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:46.923+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:46.923+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:46.923+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:46.923+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:46.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:46.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:46.856+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:47.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:46.856+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:47.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:47.315+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26798 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:47.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:47.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:47.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:47.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:47.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:47.563+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:47.950+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:47.950+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:47.950+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:47.950+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:47.873+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:47.873+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:47.873+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:47.873+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:48.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:48.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:48.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:48.561+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:48.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:48.561+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:48.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:48.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:48.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:48.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:48.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:48.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:48.898+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:48.898+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:49.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:49.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:49.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:49.602+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:50.200 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:49.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:49.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:49.913+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:49.913+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:50.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:50.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:50.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:50.201 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:50.015+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:50.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:50.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:50.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:50.568+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:51.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:51.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:51.034+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:51.034+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:50.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:50.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:50.929+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:50.929+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:51.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:51.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:51.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:51.562+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:51.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:51.562+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:52.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:52.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:52.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:52.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:52.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:52.021+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:52.321 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:52.021+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:52.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:51.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:52.321 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:51.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:52.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:51.950+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:52.322 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:51.950+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:52.582 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:52.319+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26891 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:52.583 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:52.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:52.583 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:52.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:52.583 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:52.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:52.583 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:52.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:53.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:53.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:53.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:53.019+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:52.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:52.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:52.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:52.987+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:53.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:53.528+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:53.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:53.528+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:53.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:53.528+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:53.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:53.529+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:54.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:54.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:54.060+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:54.060+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:53.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:53.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:53.961+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:53.961+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:54.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:54.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:54.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:54.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:55.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:55.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:55.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:55.060+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:54.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:54.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:54.928+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:54.928+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:55.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:55.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:55.452+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:55.452+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:56.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:56.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:56.059+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:56.059+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:55.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:55.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:55.967+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:55.967+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:56.472+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:56.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:56.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:56.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:56.473+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:57.324 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:56.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:57.325 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:56.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:57.325 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:56.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:57.325 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:56.953+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:57.325 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:57.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:57.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:57.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:57.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:57.024+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:57.326 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:57.024+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:57.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:10:57.324+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 26985 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:10:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:57.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:57.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:57.429+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:57.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:57.430+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:57.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:57.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:57.955+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:58.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:57.955+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:58.011+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:58.011+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:58.011+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:58.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:58.011+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:58.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:58.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:58.394+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:58.394+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:58.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:58.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:58.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:10:59.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:58.931+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:59.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:59.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:59.052+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:10:59.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:10:59.053+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:10:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:59.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:59.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:59.392+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:10:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:10:59.393+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:59.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:59.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:59.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:00.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:10:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:10:59.973+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:00.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:00.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:00.004+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:00.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:00.004+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:00.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:00.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:00.402+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:00.402+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:00.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:00.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:00.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:01.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:00.937+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:01.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:01.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:01.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:01.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:01.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:01.738 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:01.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:01.738 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:01.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:01.738 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:01.412+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:01.739 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:01.412+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:01.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:01.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:01.970+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:02.001 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:01.970+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:02.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:02.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:02.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:02.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:02.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:02.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:02.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:02.000+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:02.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:02.328+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27082 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:02.393+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:02.393+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:02.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:02.393+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:02.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:02.393+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:03.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.037+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:03.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:02.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:02.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:02.937+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:03.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:02.937+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:03.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:03.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:03.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:03.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:03.440+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.988+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:03.988+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:03.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:03.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:03.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:03.946+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:04.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:04.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:04.411+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:04.411+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:04.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:04.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:04.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:05.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:04.992+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:04.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:04.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:04.976+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:04.976+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:05.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:05.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:05.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:05.400+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:05.978+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:05.978+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:05.978+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:06.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:05.978+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:05.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:05.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:05.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:06.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:05.979+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:06.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:06.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:06.434+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:06.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:06.434+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:07.334 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:06.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:07.334 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:06.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:07.334 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:06.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:07.335 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:06.962+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:07.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:06.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:07.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:06.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:07.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:06.943+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:07.335 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:06.943+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:07.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:07.333+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27176 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:07.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:07.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:07.400+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:07.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:07.400+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:07.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:07.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:07.935+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:07.935+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:07.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:07.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:07.911+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:07.911+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:08.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:08.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:08.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:08.442+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:08.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:08.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:08.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:08.921+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:08.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:08.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:08.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:08.929+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:09.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:09.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:09.440+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:09.440+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:10.202 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:09.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:10.202 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:09.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:10.202 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:09.909+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:10.203 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:09.909+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:10.203 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:09.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:10.203 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:09.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:10.203 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:09.921+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:10.203 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:09.921+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:10.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:10.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:10.439+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:10.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:10.439+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:10.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:10.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:10.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:10.922+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:10.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:10.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:10.947+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:10.947+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:11.739 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:11.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:11.739 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:11.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:11.739 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:11.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:11.739 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:11.481+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:11.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:11.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:11.940+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:12.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:11.940+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:11.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:11.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:11.916+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:11.917+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:12.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:12.337+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27271 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:12.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:12.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:12.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:12.473+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:12.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:12.473+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:12.943+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:12.943+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:12.943+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:12.943+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:12.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:12.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:12.882+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:12.882+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:13.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:13.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:13.482+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:13.482+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:14.209 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:13.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:14.209 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:13.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:14.209 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:13.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:14.210 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:13.905+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:14.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:13.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:14.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:13.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:14.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:13.902+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:14.210 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:13.902+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:14.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:14.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:14.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:14.497+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:14.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:14.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:14.888+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:14.888+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:14.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:14.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:14.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:14.948+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:15.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:15.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:15.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:15.503+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:15.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:15.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:15.901+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:15.901+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:16.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:15.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:15.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:15.932+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:15.932+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:16.547+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:16.547+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:16.547+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:16.547+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:17.342 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:16.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:17.342 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:16.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:17.342 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:16.981+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:17.343 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:16.981+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:17.343 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:16.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:17.343 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:16.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:17.343 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:16.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:17.343 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:16.900+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:17.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:17.340+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27366 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:17.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:17.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:17.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:17.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:17.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:17.537+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:17.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:17.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:17.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:17.942+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:17.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:17.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:17.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:18.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:17.973+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:18.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:18.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:18.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:18.559+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:18.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:18.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:18.922+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:18.922+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:18.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:18.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:18.987+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:19.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:18.987+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:19.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:19.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:19.517+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:19.517+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:20.353 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:20.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:20.353 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:20.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:20.353 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:20.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:20.353 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:20.025+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:20.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:19.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:20.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:19.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:20.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:19.972+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:20.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:19.972+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:20.620 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:20.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:20.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:20.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:20.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:20.544+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.015+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:21.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:21.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:21.033+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:21.033+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:21.528+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:21.528+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:21.528+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:21.529+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:22.346 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:22.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:22.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.991+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:22.347 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:21.992+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:22.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:22.080+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:22.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:22.080+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:22.347 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:22.080+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:22.348 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:22.080+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:22.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:22.345+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27463 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:22.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:22.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:22.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:22.545+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:22.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:22.545+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:22.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:22.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:22.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:22.957+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:23.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:23.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:23.054+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:23.054+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:23.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:23.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:23.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:23.502+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:24.029+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:24.029+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:24.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:24.029+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:24.029+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:24.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:23.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:24.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:23.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:24.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:23.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:24.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:23.908+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:24.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:24.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:24.461+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:24.461+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:25.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:25.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:25.038+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:25.038+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:24.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:24.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:24.920+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:25.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:24.920+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:25.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:25.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:25.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:25.485+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:26.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:26.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:26.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:26.077+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:26.077+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:25.969+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:25.969+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:25.969+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:25.969+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:26.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:26.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:26.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:26.515+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:27.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:27.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:27.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:27.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:27.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:27.041+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:27.351 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:27.041+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:27.351 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:26.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:27.352 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:26.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:27.352 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:26.996+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:27.352 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:26.996+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:27.350+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27557 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:27.556+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:27.556+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:27.556+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:27.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:27.556+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:27.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:27.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:27.961+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:28.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:27.961+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:28.019+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:28.019+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:28.019+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:28.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:28.019+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:28.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:28.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:28.535+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:28.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:28.535+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:28.994+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:28.994+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:28.994+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:28.994+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:29.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:29.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:29.011+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:29.011+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:29.526+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:29.526+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:29.526+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:29.526+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:29.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:29.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:29.990+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:29.990+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:30.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:30.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:30.044+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:30.044+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:30.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:30.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:30.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:30.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:30.573+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.017+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.017+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:31.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:31.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:31.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:31.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:31.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:31.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:31.001+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:31.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:31.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:31.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:31.544+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:32.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.982+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:32.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.982+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:32.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.982+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:32.356 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:31.982+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:32.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:32.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:32.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:32.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:32.357 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:32.006+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:32.358 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:32.006+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:32.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:32.354+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27653 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:32.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:32.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:32.568+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:32.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:32.568+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.002+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:33.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:33.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:33.025+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:33.025+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:33.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:33.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:33.577+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:33.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:33.577+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:33.960+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.027+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.027+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:34.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:34.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:34.538+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:34.538+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:34.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:34.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:34.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:35.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:34.937+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.988+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:34.988+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:35.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:35.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:35.505+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:35.505+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:35.953+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:35.953+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:35.953+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:35.953+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:35.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:35.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:35.972+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:35.972+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:36.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:36.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:36.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:36.485+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:37.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:36.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:37.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:36.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:37.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:36.922+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:37.360 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:36.922+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:37.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:36.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:37.360 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:36.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:37.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:36.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:37.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:36.919+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:37.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:37.358+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27747 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:37.446+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:37.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:37.446+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:37.446+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:37.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:37.446+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:37.943+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:37.943+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:37.943+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:37.943+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:37.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:37.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:37.925+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:37.925+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:38.414+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:38.414+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:38.414+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:38.414+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:38.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:38.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:39.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:38.937+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:39.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:38.937+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:38.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:38.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:38.901+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:39.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:38.901+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:39.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:39.885 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:39.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:39.885 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:39.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:39.886 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:39.431+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:40.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:39.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:40.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:39.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:40.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:39.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:40.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:39.987+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:40.205 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:39.938+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:40.205 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:39.938+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:40.206 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:39.938+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:40.206 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:39.938+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:40.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:40.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:40.462+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:40.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:40.462+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:40.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:40.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:40.965+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:40.965+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:40.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:40.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:40.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:40.980+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:41.741 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:41.487+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:41.741 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:41.487+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:41.742 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:41.487+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:41.742 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:41.487+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:41.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:41.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:41.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:42.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:41.963+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:42.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:41.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:41.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:41.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:41.962+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:42.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:42.363+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27845 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:42.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:42.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:42.494+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:42.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:42.494+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:42.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:42.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:42.927+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:43.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:42.927+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:43.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:42.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:42.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:42.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:43.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:42.947+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:43.511+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:43.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:43.511+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:43.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:43.511+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:43.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:43.511+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:43.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:43.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:43.965+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:44.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:43.965+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:43.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:43.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:43.914+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:44.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:43.914+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:44.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:44.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:44.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:44.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:44.475+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:44.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:44.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:44.931+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:45.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:44.931+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:44.952+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:44.952+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:44.952+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:45.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:44.952+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:45.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:45.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:45.523+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:45.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:45.523+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:45.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:45.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:45.979+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:45.979+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:45.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:45.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:45.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:46.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:45.938+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:46.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:46.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:46.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:46.520+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:46.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:46.520+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:47.368 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:46.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:47.368 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:46.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:47.368 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:46.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:47.369 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:46.980+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:47.369 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:46.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:47.369 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:46.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:47.369 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:46.948+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:47.369 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:46.948+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:47.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:47.367+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 27938 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:47.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:47.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:47.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:47.481+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:47.622 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:47.481+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:47.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:47.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:47.992+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:47.992+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:47.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:47.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:47.924+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:48.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:47.924+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:48.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:48.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:48.463+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:48.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:48.464+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:48.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:49.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:48.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:48.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:48.989+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:48.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:48.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:48.939+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:48.939+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:49.444 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:49.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:49.445 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:49.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:49.445 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:49.442+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:49.445 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:49.442+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:50.030+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:50.030+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:50.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:50.030+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:50.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:50.031+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:49.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:49.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:49.918+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:50.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:49.918+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:50.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:50.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:50.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:50.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:50.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:50.480+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:50.873 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:50.480+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:51.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:51.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:51.019+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:51.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:51.019+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:50.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:50.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:50.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:51.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:50.944+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:51.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:51.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:51.509+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:51.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:51.509+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:52.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.000+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:51.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:51.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:51.956+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:52.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:51.956+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:52.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:52.372+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28034 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:52.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:52.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:52.559+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:52.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:52.559+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:53.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:52.989+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:52.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:52.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:52.962+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:53.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:52.962+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:53.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:53.533+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:53.533+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:53.533+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:53.533+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:53.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:53.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:53.989+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:54.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:53.989+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:53.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:53.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:53.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:54.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:53.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:54.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:54.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:54.496+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:54.496+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:55.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:55.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:55.032+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:55.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:55.032+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:54.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:54.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:54.921+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:55.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:54.921+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:55.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:55.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:55.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:55.495+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:56.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.026+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:56.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:55.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:55.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:55.944+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:56.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:55.944+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:56.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:56.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:56.537+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:56.537+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.993+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:57.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:56.993+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:56.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:56.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:56.983+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:57.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:56.983+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:57.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:11:57.375+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28127 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:11:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:57.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:57.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:57.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:57.543+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:57.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:57.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:57.962+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:58.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:57.962+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:57.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:57.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:57.971+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:58.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:57.971+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:58.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:58.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:58.495+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:58.495+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:11:59.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.002+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:58.949+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:58.949+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:58.949+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:11:59.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:58.949+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:11:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:59.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:59.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:59.475+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:11:59.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:11:59.475+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:00.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:11:59.980+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:59.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:59.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:59.953+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:00.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:11:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:11:59.953+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:00.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:00.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:00.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:00.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:00.493+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:00.994+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:00.994+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:00.994+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:01.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:00.994+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:00.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:00.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:00.930+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:01.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:00.930+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:01.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:01.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:01.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:01.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:01.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:01.493+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:01.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:01.493+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:02.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:02.003+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:02.003+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:02.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:02.003+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:01.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:01.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:01.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:02.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:01.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:02.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:02.380+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28225 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:02.500+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:02.500+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:02.500+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:02.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:02.500+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:03.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:03.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:03.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:03.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:03.361 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:03.042+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:03.362 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:03.042+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:03.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:02.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:03.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:02.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:03.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:02.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:03.362 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:02.978+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:03.548+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:03.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:03.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:03.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:03.549+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:04.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:04.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:04.054+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:04.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:04.054+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:04.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:03.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:03.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:03.941+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:04.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:03.941+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:04.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:04.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:04.525+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:04.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:04.525+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:05.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:05.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:05.044+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:05.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:05.044+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:05.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:04.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:04.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:04.959+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:05.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:04.959+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:05.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:05.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:05.543+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:05.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:05.543+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:06.093+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:06.093+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:06.093+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:06.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:06.094+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:05.954+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:05.954+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:05.954+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:06.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:05.954+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:06.502+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:06.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:06.503+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:06.503+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:07.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:07.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:07.068+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:07.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:07.069+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:06.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:06.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:06.968+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:07.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:06.968+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:07.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:07.385+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28319 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:07.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:07.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:07.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:07.512+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:07.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:07.512+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:08.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:08.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:08.027+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:08.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:08.028+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:07.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:07.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:07.980+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:08.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:07.981+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:08.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:08.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:08.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:08.485+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:08.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:08.485+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:09.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:09.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:09.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:09.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:09.037+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:09.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:08.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:08.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:08.978+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:09.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:08.978+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:09.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:09.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:09.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:09.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:09.476+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:09.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:09.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:09.998+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:09.998+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:10.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:10.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:10.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:10.026+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:10.026+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:10.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:10.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:10.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:10.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:10.465+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:11.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:11.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:11.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:11.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:11.051+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.016+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:11.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.016+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:11.743 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:11.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:11.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:11.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:11.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:11.488+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:11.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:11.488+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:12.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:12.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:12.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.984+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:12.037 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:11.985+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:12.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:12.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:12.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:12.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:12.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:12.036+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:12.389+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28415 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:12.448+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:12.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:12.448+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:12.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:12.448+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:12.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:12.448+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:13.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:13.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:13.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:13.071+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:12.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:12.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:12.993+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:13.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:12.993+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:13.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:13.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:13.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:13.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:13.476+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:14.022+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:14.022+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:14.022+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:14.022+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:13.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:13.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:13.946+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:14.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:13.946+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:14.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:14.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:14.432+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:14.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:14.432+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:15.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:15.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:15.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:15.043+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:15.043+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:14.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:14.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:14.945+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:15.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:14.945+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:15.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:15.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:15.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:15.478+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:16.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:16.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:16.036+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:16.036+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:15.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:15.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:16.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:15.992+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:16.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:15.992+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:16.506+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:16.506+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:16.506+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:16.506+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:17.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:17.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:17.040+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:17.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:17.040+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.000+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:17.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.001+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:17.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:17.394+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28509 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:17.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:17.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:17.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:17.513+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.973+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:18.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:17.973+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:18.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:18.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:18.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:18.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:18.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:18.071+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:18.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:18.071+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:18.497+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:18.497+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:18.497+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:18.497+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:19.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:19.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:19.041+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:19.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:19.041+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:19.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:19.010+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:19.010+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:19.010+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:19.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:19.010+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:19.531+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:19.531+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:19.531+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:19.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:19.532+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:20.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:20.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:20.050+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:20.050+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:20.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:20.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:20.049+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:20.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:20.049+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:20.540+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:20.540+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:20.540+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:20.540+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:21.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:21.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:21.014+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:21.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:21.014+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:21.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:21.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:21.064+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:21.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:21.064+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:21.566+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:21.566+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:21.566+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:21.566+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:22.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:22.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:22.037+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:22.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:22.037+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:22.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:22.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:22.100+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:22.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:22.100+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:22.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:22.398+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28607 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:22.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:22.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:22.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:22.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:22.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:22.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:22.574+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.006+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.006+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:23.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.006+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:23.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.006+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:23.123+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:23.123+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:23.123+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:23.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:23.123+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:23.592+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:23.592+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:23.592+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:23.593+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.976+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:24.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:23.976+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:24.075+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:24.075+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:24.075+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:24.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:24.075+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:24.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:24.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:24.553+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:24.553+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:24.960 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-10T07:12:24.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:12:24.968 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-10T07:12:24.969 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-10T07:12:24.982 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T07:12:24.994 INFO:tasks.cephadm:Teardown begin 2024-02-10T07:12:24.994 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-10T07:12:24.998 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:12:25.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:12:25.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:12:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:25.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:25.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:25.008+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:25.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:25.008+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:25.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:25.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:25.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:25.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:25.125+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:25.536+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:25.536+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:25.536+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:25.536+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:26.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:26.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:26.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:26.046+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:26.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:26.046+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:26.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:26.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:26.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:26.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:26.148+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:26.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:26.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:26.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:26.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:26.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:26.499+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:26.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:26.499+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:27.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:27.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:27.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:27.076+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:27.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:27.076+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:27.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:27.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:27.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:27.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:27.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:27.164+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:27.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:27.164+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:27.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:27.403+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28701 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:27.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:27.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:27.476+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:27.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:27.476+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:28.096 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:12:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:28.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:28.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:28.058+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:28.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:28.058+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:28.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:28.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:28.124+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:28.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:28.124+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:28.521+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:28.521+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:28.521+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:28.521+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:29.069+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:29.069+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:29.069+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:29.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:29.069+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:29.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:29.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:29.148+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:29.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:29.148+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:29.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:29.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:29.486+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:29.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:29.487+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:30.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:30.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:30.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:30.087+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:30.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:30.087+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:30.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:30.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:30.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:30.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:30.187+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:30.479+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:30.479+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:30.479+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:30.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:30.479+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:31.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:31.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:31.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:31.051+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:31.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:31.051+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:31.220+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:31.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:31.220+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:31.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:31.220+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:31.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:31.221+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:31.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:31.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:31.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:31.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:31.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:31.431+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:31.744 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:31.431+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:32.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:32.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:32.048+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:32.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:32.049+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:32.186+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:32.186+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:32.186+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:32.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:32.186+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:32.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:32.408+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28796 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:32.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:32.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:32.455+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:32.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:32.455+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:33.063+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:33.063+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:33.063+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:33.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:33.063+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:33.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:33.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:33.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:33.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:33.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:33.209+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:33.450+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:33.450+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:33.450+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:33.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:33.450+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:34.079+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:34.079+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:34.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:34.079+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:34.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:34.080+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:34.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:34.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:34.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:34.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:34.173+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:34.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:34.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:34.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:34.465+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:34.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:34.465+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:35.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:35.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:35.045+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:35.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:35.045+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:35.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:35.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:35.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:35.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:35.216+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:35.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:35.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:35.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:35.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:35.514+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:36.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:36.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:36.020+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:36.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:36.020+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:36.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:36.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:36.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:36.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:36.226+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:36.522+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:36.522+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:36.522+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:36.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:36.522+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:37.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:12:37.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:12:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:37.039+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:37.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:37.039+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:37.039+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:37.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:37.039+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:37.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:37.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:37.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:37.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:37.235+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:37.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:37.411+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28890 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:37.557+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:37.557+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:37.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:37.557+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:37.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:37.557+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:38.038+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:38.038+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:38.038+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:38.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:38.038+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:38.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:38.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:38.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:38.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:38.258+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:38.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:38.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:38.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:38.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:38.602+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:39.291 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:39.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:39.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:39.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:39.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:39.025+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:39.292 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:39.025+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:39.594 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:39.290+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:39.595 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:39.290+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:39.595 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:39.290+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:39.595 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:39.290+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:39.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:39.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:39.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:39.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:39.594+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:40.156 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:12:40.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:40.033+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:40.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:40.033+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:40.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:40.033+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:40.307 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:40.033+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:40.305+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:40.305+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:40.305+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:40.305+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:41.016 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:40.625+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:41.016 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:40.625+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:41.017 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:40.625+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:41.017 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:40.625+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:41.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.015+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:41.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:41.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:41.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:41.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:41.264+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:41.617+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:41.617+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:41.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:41.617+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:41.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:41.617+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:42.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:42.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:42.279 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:42.280 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:41.998+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:42.543 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:42.417+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 28987 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:42.543 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:42.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:42.544 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:42.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:42.544 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:42.278+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:42.544 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:42.278+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:42.629+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:42.629+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:42.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:42.629+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:42.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:42.629+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:43.258 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:42.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:43.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:42.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:43.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:42.985+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:43.259 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:42.986+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:43.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:43.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:43.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:43.258+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:43.972 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:43.649+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:43.972 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:43.649+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:43.972 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:43.649+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:43.972 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:43.650+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:44.255 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:43.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:44.256 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:43.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:44.256 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:43.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:44.256 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:43.971+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:44.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:44.254+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:44.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:44.254+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:44.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:44.254+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:44.602 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:44.255+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:44.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:44.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:44.601+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:44.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:44.601+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:45.222 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:44.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:45.222 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:44.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:45.222 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:44.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:45.223 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:44.929+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:45.221+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:45.221+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:45.221+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:45.221+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:45.974 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:45.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:45.974 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:45.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:45.974 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:45.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:45.974 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:45.647+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:45.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:45.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:45.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:46.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:45.973+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:46.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:46.222+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:46.222+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:46.222+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:46.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:46.222+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:47.008 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:46.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:47.008 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:46.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:47.009 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:46.670+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:47.009 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:46.671+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:47.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.007+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:47.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.008+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:47.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:47.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:47.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:47.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:47.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:47.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:47.373 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:47.173+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:47.638 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:47.422+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29081 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:47.961 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:47.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:47.961 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:47.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:47.961 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:47.637+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:47.961 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:47.637+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:48.370 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:48.149+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:48.149+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:48.149+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:48.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:48.150+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:48.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:48.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:47.960+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:48.615+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:48.615+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:48.615+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:48.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:48.615+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:49.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:49.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:49.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:49.160+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:49.372 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:49.160+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:48.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:48.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:49.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:48.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:49.373 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:48.946+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:49.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:49.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:49.605+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:49.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:49.605+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:50.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:49.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:50.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:49.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:50.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:49.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:50.182 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:49.925+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:50.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:50.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:50.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:50.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:50.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:50.181+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:50.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:50.181+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:50.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:50.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:50.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:50.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:50.599+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:51.183 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:50.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:51.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:50.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:51.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:50.929+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:51.184 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:50.929+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:51.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:51.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:51.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:51.183+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:51.916 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:51.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:51.917 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:51.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:51.917 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:51.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:51.917 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:51.636+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:52.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:12:52.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:12:52.173 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:51.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:52.174 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:51.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:52.174 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:51.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:52.174 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:51.916+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:52.427 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:52.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:52.427 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:52.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:52.427 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:52.173+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:52.428 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:52.173+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:52.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:52.426+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29177 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:52.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:52.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:52.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:52.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:52.594+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:52.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:52.594+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:53.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:52.925+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:53.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:52.925+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:53.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:52.925+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:53.205 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:52.925+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:53.598 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:53.204+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:53.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:53.204+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:53.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:53.204+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:53.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:53.204+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:53.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:53.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:53.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:53.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:53.598+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:54.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:53.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:54.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:53.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:54.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:53.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:54.211 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:53.932+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:54.549 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:54.210+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:54.550 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:54.210+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:54.550 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:54.210+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:54.550 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:54.210+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:54.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:54.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:54.549+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:54.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:54.549+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:55.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:54.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:55.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:54.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:55.214 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:54.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:55.215 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:54.910+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:55.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:12:55.572 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:55.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:55.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:55.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:55.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:55.213+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:55.573 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:55.213+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:55.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:55.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:55.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:55.572+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:55.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:55.572+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:56.233 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:55.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:56.233 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:55.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:56.233 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:55.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:56.234 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:55.931+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:56.567 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:56.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:56.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:56.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:56.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:56.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:56.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:56.233+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:56.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:56.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:56.567+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:56.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:56.567+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:57.189 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:56.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:57.189 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:56.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:57.189 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:56.931+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:57.189 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:56.931+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:57.445 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:12:57.431+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29271 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:12:57.445 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:57.188+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:57.445 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:57.188+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:57.446 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:57.188+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:57.446 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:57.188+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:57.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:57.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:57.586+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:57.586+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:58.218 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:57.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:58.218 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:57.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:58.218 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:57.928+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:58.218 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:57.928+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:58.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:12:58.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:12:58.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:12:58.300 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:12:58.301 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T07:12:58.301 DEBUG:teuthology.orchestra.run.smithi073:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:12:58.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:12:58.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:12:58.604 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:58.217+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:58.604 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:58.217+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:58.604 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:58.217+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:58.604 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:58.217+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:58.603+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:58.603+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:58.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:58.603+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:58.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:58.603+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:59.252 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:58.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:59.253 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:58.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:59.253 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:58.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:12:59.253 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:58.938+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:59.252+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:59.252+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:59.252+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:12:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:12:59.252+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:12:59.890 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:59.620+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:59.890 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:59.620+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:59.890 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:59.620+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:12:59.890 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:12:59.620+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:00.301 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:59.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:00.301 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:59.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:00.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:59.890+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:00.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:12:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:12:59.890+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:00.597 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:00.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:00.597 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:00.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:00.597 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:00.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:00.597 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:00.301+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:00.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:00.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:00.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:00.596+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:00.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:00.596+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:01.294 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:00.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:01.294 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:00.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:01.295 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:00.916+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:01.295 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:00.916+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:01.376 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:13:01.620 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:01.293+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:01.293+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:01.293+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:01.294+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:01.940 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:01.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:01.940 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:01.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:01.940 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:01.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:01.940 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:01.630+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:02.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:01.939+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:02.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:01.939+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:02.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:01.939+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:02.289 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:01.939+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:02.562 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:02.436+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29369 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:02.562 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:02.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:02.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:02.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:02.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:02.288+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:02.563 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:02.288+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:02.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:02.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:02.655+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:02.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:02.655+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:03.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:02.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:03.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:02.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:03.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:02.907+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:03.263 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:02.907+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:03.262+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:03.262+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:03.262+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:03.262+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:03.875+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:03.875+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:03.875+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:03.875+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:04.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:03.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:04.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:03.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:04.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:03.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:04.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:03.646+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:04.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:04.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:04.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:04.269+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:04.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:05.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:04.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:05.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:04.621+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:05.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:04.621+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:04.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:04.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:05.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:04.841+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:05.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:04.841+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:05.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:05.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:05.244+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:05.244+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:05.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:05.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:05.640+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:06.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:05.640+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:05.814+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:05.814+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:05.814+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:06.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:05.814+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:06.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:06.208+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:06.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:06.208+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:06.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:06.208+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:06.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:06.208+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:06.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:06.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:06.602+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:06.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:06.602+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:06.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:06.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:06.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:06.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:06.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:06.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:06.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:06.792+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:07.567 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:07.440+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29463 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:07.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:07.221+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:07.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:07.221+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:07.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:07.221+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:07.568 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:07.221+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:07.808+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:07.808+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:07.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:07.808+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:07.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:07.808+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:07.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:07.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:07.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:07.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:07.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:07.598+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:07.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:07.598+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:08.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:08.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:08.209+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:08.209+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:08.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:09.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:08.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:09.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:08.646+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:09.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:08.646+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:09.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:08.772+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:09.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:08.772+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:09.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:08.772+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:09.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:08.772+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:09.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:09.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:09.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:09.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:09.183+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:09.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:09.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:09.743+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:10.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:09.743+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:10.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:09.685+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:10.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:09.685+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:10.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:09.685+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:10.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:09.685+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:10.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:13:10.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:10.178+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:10.178+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:10.178+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:10.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:10.178+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:10.767+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:10.767+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:10.767+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:11.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:10.767+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:11.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:10.642+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:11.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:10.642+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:11.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:10.642+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:11.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:10.642+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:11.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:11.220+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:11.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:11.220+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:11.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:11.220+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:11.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:11.220+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:11.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:11.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:11.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:11.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:11.600+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:11.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:11.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:11.754+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:11.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:11.754+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:12.571 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:12.445+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29553 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:12.571 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:12.253+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:12.571 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:12.253+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:12.571 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:12.253+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:12.571 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:12.253+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:12.627+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:12.627+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:12.627+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:12.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:12.627+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:12.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:12.780+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:12.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:12.780+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:12.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:12.780+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:12.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:12.780+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:13.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:13:13.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:13.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:13.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:13.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:13.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:13.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:13.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:13.231+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:14.017 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:13.622+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:14.017 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:13.622+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:14.017 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:13.622+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:14.018 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:13.622+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:14.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:13.732+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:14.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:13.732+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:14.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:13.732+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:14.018 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:13.732+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:14.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:14.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:14.194+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:14.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:14.194+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:14.635+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:14.635+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:15.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:14.635+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:15.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:14.635+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:14.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:14.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:14.730+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:15.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:14.730+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:15.598 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:15.170+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:15.598 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:15.170+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:15.598 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:15.170+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:15.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:15.170+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:15.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:15.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:15.597+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:15.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:15.597+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:15.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:15.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:15.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:15.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:15.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:15.716+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:15.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:15.716+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:16.574 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:16.159+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:16.574 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:16.159+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:16.575 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:16.159+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:16.575 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:16.159+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:16.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:16.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:16.573+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:16.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:16.573+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:16.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:16.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:16.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:16.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:16.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:16.707+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:16.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:16.707+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:17.450 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:17.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:17.451 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:17.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:17.451 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:17.134+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:17.451 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:17.134+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:17.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:17.449+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29647 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:17.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:17.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:17.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:17.529+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:17.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:17.529+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:17.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:17.688+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:17.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:17.688+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:17.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:17.688+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:17.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:17.688+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:18.515 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:18.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:18.516 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:18.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:18.516 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:18.163+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:18.516 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:18.163+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:18.513+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:18.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:18.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:18.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:18.514+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:18.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:18.514+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:18.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:18.738+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:18.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:18.738+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:18.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:18.738+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:18.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:18.738+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:19.479 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:19.195+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:19.480 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:19.195+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:19.480 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:19.195+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:19.480 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:19.195+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:19.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:19.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:19.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:19.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:19.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:19.478+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:19.730 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:19.479+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:19.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:19.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:19.729+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:20.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:19.729+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:20.523 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:20.197+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:20.523 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:20.197+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:20.523 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:20.197+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:20.524 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:20.197+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:20.522+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:20.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:20.522+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:20.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:20.522+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:20.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:20.522+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:20.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:20.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:20.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:20.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:20.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:20.768+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:20.873 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:20.768+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:21.545 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:21.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:21.545 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:21.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:21.545 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:21.187+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:21.546 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:21.187+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:21.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:21.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:21.544+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:21.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:21.544+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:21.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:21.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:21.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:21.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:21.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:21.724+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:21.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:21.724+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:22.455 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:22.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:22.456 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:22.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:22.456 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:22.183+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:22.456 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:22.183+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:22.760 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:22.454+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29746 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:22.761 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:22.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:22.761 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:22.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:22.761 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:22.574+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:22.761 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:22.574+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:22.760+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:22.760+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:22.760+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:22.760+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:23.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:23.232+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:23.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:23.232+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:23.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:23.232+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:23.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:23.232+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:23.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:23.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:23.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:23.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:23.599+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:23.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:23.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:23.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:23.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:23.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:23.712+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:23.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:23.713+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:24.582 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:24.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:24.582 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:24.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:24.582 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:24.231+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:24.583 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:24.231+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:24.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:24.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:24.581+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:24.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:24.581+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:24.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:24.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:24.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:24.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:24.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:24.746+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:24.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:24.746+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:25.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:13:25.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:25.600 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:25.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:25.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:25.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:25.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:25.258+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:25.601 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:25.258+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:25.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:25.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:25.600+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:25.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:25.600+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:25.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:25.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:25.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:25.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:25.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:25.706+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:25.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:25.706+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:26.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:26.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:26.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:26.237+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:27.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:26.645+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:27.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:26.645+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:27.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:26.645+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:27.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:26.645+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:26.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:26.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:26.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:27.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:26.752+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:27.586 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:27.459+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29840 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:27.587 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:27.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:27.587 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:27.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:27.587 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:27.259+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:27.587 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:27.259+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:27.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:27.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:27.599+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:27.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:27.599+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:27.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:27.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:27.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:27.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:27.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:27.752+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:27.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:27.752+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:28.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:13:28.590 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:28.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:28.590 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:28.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:28.590 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:28.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:28.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:28.270+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:28.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:28.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:28.589+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:28.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:28.589+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:28.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:28.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:28.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:28.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:28.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:28.792+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:28.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:28.792+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:29.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:29.303+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:29.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:29.303+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:29.592 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:29.303+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:29.592 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:29.303+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:29.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:29.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:29.590+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:29.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:29.590+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:29.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:29.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:29.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:29.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:29.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:29.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:29.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:29.815+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:30.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:30.308+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:30.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:30.308+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:30.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:30.308+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:30.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:30.308+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:30.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:30.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:31.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:30.636+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:31.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:30.636+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:30.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:30.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:30.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:31.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:30.862+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:31.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:13:31.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:31.580 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:13:31.580 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:13:31.581 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T07:13:31.582 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144/remote/smithi073/crash 2024-02-10T07:13:31.583 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:13:31.612 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:31.287+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:31.612 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:31.287+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:31.613 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:31.287+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:31.613 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:31.287+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:31.616 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144/remote/smithi181/crash 2024-02-10T07:13:31.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:13:31.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:31.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:31.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:31.630+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:31.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:31.631+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:32.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:31.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:32.303 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:31.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:32.303 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:31.899+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:32.303 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:31.899+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:32.590 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:32.463+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 29937 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:32.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:32.302+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:32.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:32.302+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:32.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:32.302+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:32.591 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:32.302+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:32.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:32.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:32.618+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:32.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:32.618+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:33.263 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:32.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:33.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:32.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:33.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:32.942+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:33.264 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:32.942+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:33.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:33.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:33.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:33.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:33.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:33.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:33.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:33.263+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:33.980 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:33.656+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:33.981 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:33.656+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:33.981 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:33.656+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:33.981 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:33.657+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:34.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:33.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:34.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:33.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:34.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:33.980+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:34.302 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:33.980+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:34.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:34.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:34.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:34.301+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:34.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:13:35.015 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:34.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:35.016 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:34.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:35.016 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:34.639+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:35.016 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:34.639+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:35.270 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:35.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:35.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.015+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:35.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.015+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:35.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:35.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:35.269+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:35.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:35.269+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:35.988 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:35.683+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:35.988 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:35.683+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:35.988 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:35.683+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:35.988 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:35.683+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:36.297 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:36.298 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:36.298 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.987+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:36.298 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:35.987+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:36.297+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:36.297+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:36.297+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:36.297+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:37.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:36.697+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:37.001 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:36.697+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:37.002 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:36.697+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:37.002 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:36.697+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:37.287 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:37.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:37.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.000+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:37.288 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.000+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:37.592 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:37.468+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30031 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:37.592 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:37.287+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:37.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:37.287+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:37.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:37.287+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:37.593 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:37.287+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:37.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:37.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:37.725+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:37.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:37.725+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:38.329 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:38.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:38.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.973+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:38.330 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:37.973+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:38.329+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:38.329+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:38.329+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:38.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:38.329+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:38.760+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:38.760+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:38.760+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:39.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:38.760+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:38.966+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:38.966+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:38.966+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:39.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:38.966+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:39.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:39.359+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:39.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:39.359+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:39.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:39.359+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:39.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:39.359+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:39.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:39.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:39.960+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:40.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:39.960+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:40.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:39.768+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:40.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:39.768+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:40.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:39.768+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:40.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:39.769+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:40.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:40.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:40.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:40.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:40.331+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:40.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:40.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:40.957+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:41.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:40.957+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:41.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:40.769+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:41.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:40.769+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:41.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:40.769+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:41.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:40.769+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:41.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:41.318+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:41.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:41.318+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:41.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:41.318+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:41.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:41.319+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:41.753+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:41.753+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:42.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:41.753+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:42.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:41.754+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:41.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:41.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:41.998+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:42.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:41.999+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:42.598 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:42.473+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30130 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:42.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:42.309+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:42.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:42.309+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:42.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:42.309+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:42.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:42.309+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:42.870 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:42.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:42.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:42.715+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:42.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:42.715+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:43.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:42.977+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:43.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:42.977+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:43.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:42.977+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:43.332 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:42.977+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:43.332+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:43.332+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:43.332+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:43.332+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:43.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:13:43.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:43.717+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:43.717+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:43.717+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:44.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:43.717+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:43.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:43.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:43.932+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:44.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:43.933+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:44.361+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:44.361+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:44.361+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:44.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:44.361+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:44.766+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:44.766+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:44.766+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:45.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:44.766+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:44.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:44.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:44.959+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:45.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:44.959+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:45.740 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:45.385+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:45.740 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:45.385+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:45.741 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:45.385+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:45.741 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:45.385+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:45.739+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:45.739+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:45.739+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:46.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:45.739+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:45.941+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:45.941+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:45.941+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:46.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:45.941+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:46.365+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:46.365+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:46.365+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:46.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:46.365+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:46.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:13:47.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:46.705+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:47.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:46.705+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:47.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:46.705+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:47.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:46.705+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:46.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:47.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:46.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:47.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:46.919+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:47.123 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:46.919+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:47.602 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:47.478+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30224 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:47.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:47.351+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:47.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:47.351+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:47.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:47.351+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:47.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:47.351+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:47.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:47.743+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:47.880 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:47.743+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:47.880 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:47.743+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:47.880 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:47.743+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:48.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:47.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:48.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:47.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:48.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:47.938+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:48.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:47.938+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:48.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:48.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:48.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:48.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:48.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:48.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:48.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:48.340+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:49.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:48.915+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:49.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:48.915+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:49.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:48.915+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:49.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:48.915+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:49.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:48.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:49.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:48.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:49.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:48.695+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:49.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:48.695+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:49.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:49.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:49.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:49.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:49.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:49.340+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:49.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:49.340+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:49.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:49.690+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:49.947 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:49.690+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:49.947 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:49.690+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:49.947 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:49.690+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:50.354 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:49.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:50.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:49.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:50.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:49.946+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:50.355 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:49.946+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:50.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:50.354+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:50.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:50.354+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:50.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:50.354+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:50.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:50.354+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:50.696+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:50.696+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:50.696+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:51.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:50.696+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:50.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:50.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:50.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:51.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:50.910+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:51.359+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:51.359+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:51.359+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:51.359+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:51.717+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:51.717+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:51.717+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:52.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:51.717+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:51.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:51.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:51.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:51.900+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:52.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:52.483+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30321 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:52.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:52.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:52.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:52.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:52.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:52.368+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:52.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:52.368+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:52.729+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:52.729+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:52.729+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:52.729+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:52.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:52.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:52.863+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:52.863+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:53.327+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:53.328+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:53.328+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:53.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:53.328+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:53.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:53.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:54.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:53.714+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:54.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:53.714+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:53.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:53.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:53.840+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:54.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:53.840+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:54.307+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:54.307+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:54.307+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:54.307+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:54.760+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:54.760+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:54.760+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:55.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:54.760+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:54.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:54.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:54.815+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:55.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:54.815+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:55.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:55.345+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:55.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:55.345+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:55.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:55.345+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:55.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:55.345+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:55.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:55.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:55.758+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:56.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:55.758+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:55.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:55.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:55.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:56.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:55.821+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:56.319+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:56.319+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:56.319+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:56.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:56.319+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:56.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:56.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:56.821+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:56.821+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:57.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:56.792+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:57.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:56.792+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:57.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:56.792+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:57.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:56.793+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:57.612 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:13:57.486+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30415 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:13:57.612 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:57.293+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:57.613 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:57.293+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:57.613 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:57.293+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:57.613 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:57.294+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:57.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:57.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:57.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:57.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:57.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:57.806+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:57.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:57.806+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:57.793+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:57.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:57.793+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:57.793+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:57.872 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:57.793+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:58.313+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:58.313+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:58.313+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:58.313+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:58.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:13:58.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:58.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:58.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:58.838+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:13:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:58.838+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:59.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:58.831+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:58.831+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:58.831+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:13:59.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:58.831+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:13:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:59.357+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:59.357+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:59.357+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:13:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:13:59.357+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:59.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:59.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:59.832+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:13:59.832+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:59.867+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:59.867+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:59.867+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:00.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:13:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:13:59.867+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:00.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:00.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:00.331+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:00.332+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:00.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:00.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:00.786+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:00.786+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:01.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:00.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:01.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:00.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:01.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:00.888+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:01.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:00.888+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:01.750 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:01.378+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:01.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:01.378+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:01.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:01.378+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:01.751 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:01.379+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:01.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:14:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:01.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:01.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:02.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:01.773+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:02.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:01.773+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:02.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:01.923+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:02.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:01.923+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:02.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:01.923+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:02.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:01.923+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:02.618 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:02.492+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30514 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:02.618 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:02.363+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:02.618 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:02.363+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:02.618 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:02.363+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:02.619 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:02.363+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:02.813+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:02.813+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:02.813+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:02.813+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:03.169 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:02.907+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:03.169 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:02.907+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:03.169 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:02.907+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:03.169 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:02.907+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:03.374+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:03.374+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:03.374+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:03.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:03.375+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:03.949+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:03.949+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:04.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:03.949+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:04.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:03.949+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:03.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:03.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:03.801+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:04.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:03.801+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:04.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:14:04.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:04.838 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:04.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:04.838 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:04.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:04.838 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:04.404+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:04.838 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:04.404+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:04.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:14:04.892 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:14:04.893 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T07:14:04.893 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-10T07:14:04.934 INFO:teuthology.orchestra.run.smithi073.stdout:1707547265.6486607 mon.smithi073 (mon.0) 719 : cluster [WRN] Replacing daemon mds.cephfs.smithi073.nuvejb as rank 0 with standby daemon mds.cephfs.smithi181.cuebex 2024-02-10T07:14:04.934 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T07:14:04.935 INFO:tasks.cephadm:Compressing logs... 2024-02-10T07:14:04.935 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:14:04.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:14:04.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:05.005 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:14:05.005 INFO:teuthology.orchestra.run.smithi073.stderr:: No such file or directory 2024-02-10T07:14:05.005 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi073.ykxzap.log 2024-02-10T07:14:05.008 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi073.vjivfn.log 2024-02-10T07:14:05.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:14:05.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi073.ykxzap.log: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi073.vjivfn.log: gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T07:14:05.009 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi073.nuvejb.log 2024-02-10T07:14:05.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.audit.log: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T07:14:05.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi073.nuvejb.log: gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:14:05.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:14:05.010 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log: 90.0% 81.6% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:14:05.010 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:14:05.010 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T07:14:05.011 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:14:05.011 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log: 86.0% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:14:05.014 INFO:teuthology.orchestra.run.smithi073.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:14:05.014 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi073.log 2024-02-10T07:14:05.026 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi073.log: 90.9% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:14:05.031 INFO:teuthology.orchestra.run.smithi073.stderr: 95.8% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi073.ykxzap.log.gz 2024-02-10T07:14:05.042 INFO:teuthology.orchestra.run.smithi073.stderr: 94.9% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi073.nuvejb.log.gz 2024-02-10T07:14:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:04.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:05.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:04.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:04.837+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:05.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:04.837+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:05.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:04.942+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:05.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:04.942+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:05.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:04.942+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:05.123 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:04.942+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:05.776 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi073.vjivfn.log: file size changed while zipping 2024-02-10T07:14:05.777 INFO:teuthology.orchestra.run.smithi073.stderr: 90.0% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi073.vjivfn.log.gz 2024-02-10T07:14:05.859 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:05.408+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:05.859 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:05.408+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:05.859 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:05.408+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:05.860 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:05.408+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:05.904 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi073.log: file size changed while zipping 2024-02-10T07:14:05.904 INFO:teuthology.orchestra.run.smithi073.stderr: 92.6% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi073.log.gz 2024-02-10T07:14:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:05.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:05.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:05.858+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:06.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:05.858+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:06.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:05.910+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:06.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:05.910+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:06.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:05.910+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:06.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:05.910+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:06.132 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-10T07:14:06.132 INFO:teuthology.orchestra.run.smithi073.stderr: 93.3% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T07:14:06.401 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-10T07:14:06.402 INFO:teuthology.orchestra.run.smithi073.stderr: 93.3% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T07:14:06.629 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-10T07:14:06.629 INFO:teuthology.orchestra.run.smithi073.stderr: 93.2% -- replaced with /var/log/ceph/3c2b87a2-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T07:14:06.630 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-02-10T07:14:06.630 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m1.640s 2024-02-10T07:14:06.631 INFO:teuthology.orchestra.run.smithi073.stderr:user 0m5.770s 2024-02-10T07:14:06.631 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m0.138s 2024-02-10T07:14:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:06.439+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:06.439+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:06.439+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:06.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:06.439+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:06.946+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:06.946+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:06.946+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:07.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:06.946+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:06.878+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:06.878+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:06.878+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:07.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:06.878+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:07.832 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:07.497+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30608 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:07.833 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:07.418+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:07.833 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:07.418+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:07.833 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:07.418+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:07.833 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:07.418+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:08.028 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:14:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:07.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:07.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:07.970+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:08.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:07.970+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:07.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:07.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:07.831+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:08.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:07.831+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:08.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:08.460+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:08.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:08.460+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:08.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:08.460+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:08.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:08.460+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.014+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.014+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:09.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.014+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:09.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.014+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:08.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:08.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:08.877+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:09.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:08.877+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:09.449+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:09.449+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:09.449+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:09.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:09.449+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.988+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.988+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.988+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:10.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:09.988+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:09.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:09.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:09.882+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:10.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:09 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:09.883+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:10.426+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:10.426+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:10.426+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:10.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:10.426+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:11.038+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:11.038+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:11.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:11.038+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:11.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:11.038+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:10.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:10.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:10.904+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:11.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:10 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:10.904+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:11.752 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:11.411+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:11.752 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:11.411+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:11.752 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:11.411+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:11.752 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:11.411+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:12.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:11.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:12.015 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:11.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:12.016 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:11.874+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:12.016 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:11 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:11.874+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:12.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:12.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:12.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:12.015+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:12.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:12.015+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:12.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:12.502+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30705 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:12.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:12.458+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:12.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:12.458+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:12.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:12.458+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:12.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:12.458+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:12.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:12.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:12.880+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:13.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:12 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:12.880+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.033+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.033+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.033+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:13.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.033+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:13.454+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:13.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:13.454+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:13.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:13.454+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:13.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:13.454+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:13.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:13.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:13.900+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:14.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:13.900+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.994+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.994+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.994+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:14.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:13 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:13.994+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:14.486+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:14.486+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:14.486+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:14.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:14.486+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:14.986+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:14.986+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:14.986+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:15.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:14.986+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:14.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:14.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:14.894+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:15.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:14 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:14.894+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:15.455+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:15.455+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:15.455+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:15.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:15.455+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:15.941+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:15.941+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:15.941+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:16.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:15.941+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:16.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:15.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:15.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:15.910+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:16.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:15 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:15.910+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:16.460+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:16.460+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:16.460+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:16.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:16.461+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:17.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:14:17.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:16.950+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:16.950+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:16.950+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:17.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:16.950+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:16.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:16.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:16.927+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:17.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:16 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:16.927+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:17.633 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:17.506+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30799 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:17.634 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:17.458+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:17.634 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:17.458+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:17.634 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:17.458+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:17.634 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:17.458+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:17.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:17.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:18.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:17.905+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:18.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:17.905+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:17.908+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:17.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:17.909+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:18.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:17 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:17.909+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:18.426+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:18.426+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:18.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:18.426+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:18.872 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:18.426+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:18.887+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:18.887+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:19.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:18.887+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:19.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:18.887+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:18.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:18.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:18.879+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:48.324303+0000 (oldest deadline 2024-02-10T07:04:08.324303+0000) 2024-02-10T07:14:19.372 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:18 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:18.879+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:19.443+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:19.443+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:19.443+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:19.871 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:19.443+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:20.096 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:14:20.371 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:19.917+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:19.898+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:19.898+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:20.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:19.898+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:20.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:19 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:19.898+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:20.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:20.455+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:20.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:20.455+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:20.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:20.455+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:20.870 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:20.455+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:21.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:20.869+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:21.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:20.898+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:21.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:20.898+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:21.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:20.898+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:21.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:20 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:20.898+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:21.753 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:21.408+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:21.753 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:21.408+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:21.753 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:21.408+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:21.753 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:21.408+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:22.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:21.883+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:21.917+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:21.917+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:21.917+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:22.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:21 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:21.917+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:22.843 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:22.512+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30898 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:22.843 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:22.437+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:22.843 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:22.437+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:22.843 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:22.437+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:22.843 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:22.437+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:23.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:22.842+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:22.952+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:22.952+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:22.952+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:23.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:22 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:22.953+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:23.837 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:23.422+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:23.837 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:23.422+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:23.837 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:23.422+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:23.838 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:23.422+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:24.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:23.837+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:23.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:23.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:24.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:23.959+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:24.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:23 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:23.959+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:24.673 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:24.376+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:24.673 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:24.376+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:24.673 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:24.376+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:24.674 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:24.376+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:24.975 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:24.872+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:25.329 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:24.975+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:25.329 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:24.975+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:25.329 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:24.975+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:25.330 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:24 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:24.975+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:25.328+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:25.328+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:25.328+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:25.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:25.328+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:26.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:25.833+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:25.983+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:26.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:25.983+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:25.983+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:26.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:25 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:25.983+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:26.284+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:26.284+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:26.284+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:26.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:26.284+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:27.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:26.811+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:27.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:26.944+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:27.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:26.944+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:26.944+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:27.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:26 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:26.944+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:27.621 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:27.517+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 30992 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:27.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:27.307+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:27.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:27.307+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:27.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:27.307+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:27.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:27.307+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:27.993 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:27.812+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:28.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:27.993+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:28.271 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:27.993+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:28.271 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:27.993+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:28.271 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:27 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:27.993+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:28.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:28.270+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:28.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:28.270+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:28.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:28.270+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:55.932694+0000 (oldest deadline 2024-02-10T07:04:15.932694+0000) 2024-02-10T07:14:28.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:28.270+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:29.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:28.857+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:28.988+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:29.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:28.988+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:28.988+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:29.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:28 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:28.988+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:29.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:29.251+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:30.278 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:29 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:29.904+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:30.278 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:30.014+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:30.278 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:30.014+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:30.278 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:30.014+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:30.279 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:30.014+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:30.620 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:30.277+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:31.283 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:30 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:30.879+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:31.283 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:31.023+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:31.283 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:31.023+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:31.283 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:31.023+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:31.284 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:31.023+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:31.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:31.282+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:32.024 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:31 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:31.836+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:32.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:14:32.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:32.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:32.247+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:32.024+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:32.024+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:32.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:32.024+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:32.372 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:32.024+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:32.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:32.522+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 31089 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:33.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:32 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:32.852+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:33.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:33.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:33.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:33.057+0000 7feb5d2ca700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:03:59.161190+0000 (oldest deadline 2024-02-10T07:04:19.161190+0000) 2024-02-10T07:14:33.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:33.057+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:33.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:33.270+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:34.306 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:34.060+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:34.306 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:33 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:33.876+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:34.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:34.304+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:35.093 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:34 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:34.832+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:35.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:14:35.371 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:35.092+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:35.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:35.274+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:36.125 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:35 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:35.871+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:36.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:36.124+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:36.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:36.235+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:37.281 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:36 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:36.918+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:37.281 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:37.131+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:37.540 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:37.527+0000 7f6f4fb64700 -1 mon.smithi073@0(probing) e2 get_health_metrics reporting 31183 slow ops, oldest is log(1 entries from seq 289 at 2024-02-10T06:42:43.502127+0000) 2024-02-10T07:14:37.541 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:37.280+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:38.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:14:38.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:38.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:14:38.240 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:14:38.241 INFO:tasks.cephadm:Removing cluster... 2024-02-10T07:14:38.241 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 --force 2024-02-10T07:14:38.270 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:38.101+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:38.271 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:37 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:37.955+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:38.521 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:38.270+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:38 smithi073 systemd[1]: Stopping Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:14:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:38.767+0000 7f6f52b6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi073 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:14:38.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073[29059]: 2024-02-10T07:14:38.767+0000 7f6f52b6a700 -1 mon.smithi073@0(probing) e2 *** Got Signal Terminated *** 2024-02-10T07:14:39.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:39 smithi073 podman[82791]: 2024-02-10 07:14:39.063995565 +0000 UTC m=+0.401906480 container died 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-10T07:14:39.304 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:39.089+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:39.304 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:38 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:38.972+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:39.594 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:39 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:39.303+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:39 smithi073 podman[82791]: 2024-02-10 07:14:39.594931634 +0000 UTC m=+0.932842565 container cleanup 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T07:14:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:39 smithi073 bash[82791]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073 2024-02-10T07:14:39.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:39 smithi073 podman[82805]: 2024-02-10 07:14:39.779891942 +0000 UTC m=+0.713640370 container remove 9743a7ae51ad457163324ed023ec5b13c17489e8240090afa56f7522af8b92c9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-mon-smithi073, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:14:40.273 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:40.054+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:40.273 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:40 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service: Deactivated successfully. 2024-02-10T07:14:40.273 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:40 smithi073 systemd[1]: Stopped Ceph mon.smithi073 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T07:14:40.274 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Feb 10 07:14:40 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@mon.smithi073.service: Consumed 24.385s CPU time. 2024-02-10T07:14:40.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:40.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:40.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:40.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:40.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:40.002+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:40.274 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:40.002+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:40.532 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:40 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:40.272+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:41.120 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:41.018+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.047+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:41.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.047+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:41.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:41.300+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:42.340 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:42.011+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:42.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:42.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:42.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.997+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:42.341 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:41 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:41.997+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:42.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:42.339+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:43.261 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:42.964+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:43.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:42.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:43.261 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:42.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:43.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:42.963+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:43.262 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:42 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:42.963+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:43.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:43.308+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:44.190 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:43.978+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:44.190 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:43.934+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:44.190 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:43.934+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:44.190 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:43.934+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:44.190 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:43 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:43.935+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:44.499 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:44.308+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:45.167 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:44.977+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:45.167 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:44.949+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:45.168 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:44.949+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:45.168 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:44.949+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:45.168 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:44 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:44.949+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:45.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:45.335+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:46.180 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:46.010+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:46.180 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:45.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:46.181 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:45.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:46.181 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:45.924+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:46.181 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:45 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:45.924+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:46.443 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:46.360+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:47.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:47.013+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:46.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:47.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:46.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:46.971+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:47.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:46 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:46.971+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:47.535 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:47.363+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:47.952+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:47.952+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:47.952+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:48.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:47 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:47.953+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:47 smithi073 systemd[1]: Stopping Ceph osd.0 for 3c2b87a2-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:14:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:48.008+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:48.105+0000 7feb61cce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:14:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:48.105+0000 7feb61cce700 -1 osd.0 47 *** Got signal Terminated *** 2024-02-10T07:14:48.122 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:48.105+0000 7feb61cce700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:14:48.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:48.362+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:49.353 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:49.003+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:49.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:48.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:49.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:48.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:49.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:48.908+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:49.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:48 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:48.908+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:49.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:49.352+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:50.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:49.989+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:49.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:49.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:50.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:49.862+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:50.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:49 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:49.863+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:50.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:50.344+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:51.033 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:50.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:51.034 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:50.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:51.034 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:50.854+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:51.034 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:50 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:50.854+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:51.323 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:51.033+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:51.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:51.322+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:52.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:51.992+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:52.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:51.836+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:52.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:51.836+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:51.836+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:52.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:51 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:51.836+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:52.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:52.297+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:53.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0[47499]: 2024-02-10T07:14:52.959+0000 7feb5d2ca700 -1 osd.0 47 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.24273.0:4415 3.1b 3:dc3c8e75:::10000004626.00000000:head [write 0~3433 [1@-1] in=3433b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:52.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:53.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:52.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:52.845+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:53.122 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:52 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:52.845+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:53.306+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:53.306+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:53.306+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:53.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:53.306+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:53.932 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:53 smithi073 podman[83634]: 2024-02-10 07:14:53.667328184 +0000 UTC m=+5.662502514 container died 9218e1a18fa5c1046112794dba6ab306939c1f42b44448de797f07b1c3c19202 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0, RELEASE=quincy-72ce7c1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T07:14:53.932 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:53.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:53.932 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:53.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:53.932 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:53.830+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:53.933 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:53 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:53.830+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:54.280 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83634]: 2024-02-10 07:14:54.03922733 +0000 UTC m=+6.034401660 container cleanup 9218e1a18fa5c1046112794dba6ab306939c1f42b44448de797f07b1c3c19202 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T07:14:54.280 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 bash[83634]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0 2024-02-10T07:14:54.621 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83656]: 2024-02-10 07:14:54.324001502 +0000 UTC m=+0.650052089 container remove 9218e1a18fa5c1046112794dba6ab306939c1f42b44448de797f07b1c3c19202 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T07:14:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:54.279+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:54.279+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:54.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:54.279+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:54.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:54.279+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:54.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:54.790+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:54.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:54.790+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:54.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:54.790+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:54.945 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:54 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:54.791+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:54.945 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83750]: 2024-02-10 07:14:54.54818014 +0000 UTC m=+0.018076881 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-10T07:14:54.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83750]: 2024-02-10 07:14:54.666443924 +0000 UTC m=+0.136340646 container create 0980d24459fe62e3598af8a5e353ba887dce7092840616f4ff23ca337036ea92 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T07:14:54.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83750]: 2024-02-10 07:14:54.869538141 +0000 UTC m=+0.339434877 container init 0980d24459fe62e3598af8a5e353ba887dce7092840616f4ff23ca337036ea92 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, RELEASE=quincy-72ce7c1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T07:14:54.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83750]: 2024-02-10 07:14:54.872526598 +0000 UTC m=+0.342423328 container start 0980d24459fe62e3598af8a5e353ba887dce7092840616f4ff23ca337036ea92 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-10T07:14:54.946 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:54 smithi073 podman[83750]: 2024-02-10 07:14:54.944410009 +0000 UTC m=+0.414306744 container attach 0980d24459fe62e3598af8a5e353ba887dce7092840616f4ff23ca337036ea92 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:14:55.239 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:55 smithi073 podman[83750]: 2024-02-10 07:14:55.15619839 +0000 UTC m=+0.626095112 container died 0980d24459fe62e3598af8a5e353ba887dce7092840616f4ff23ca337036ea92 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:14:55.598 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:55.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:55.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:55.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:55.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:55.237+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:55.599 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:55.237+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:55 smithi073 podman[83775]: 2024-02-10 07:14:55.765174597 +0000 UTC m=+0.600371960 container remove 0980d24459fe62e3598af8a5e353ba887dce7092840616f4ff23ca337036ea92 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T07:14:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:55 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T07:14:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:55 smithi073 systemd[1]: Stopped Ceph osd.0 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T07:14:55.871 INFO:journalctl@ceph.osd.0.smithi073.stdout:Feb 10 07:14:55 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.0.service: Consumed 20.804s CPU time. 2024-02-10T07:14:55.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:55.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:55.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:55.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:55.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:55.748+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:55.872 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:55 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:55.748+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:56.252 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:55 smithi073 systemd[1]: Stopping Ceph osd.1 for 3c2b87a2-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:14:56.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:56.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:56.252 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:56.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:56.253 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:56.226+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:56.253 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:56.226+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.251+0000 7f725754f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:14:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.251+0000 7f725754f700 -1 osd.1 47 *** Got signal Terminated *** 2024-02-10T07:14:56.621 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.251+0000 7f725754f700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:14:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.727+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:57.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:56 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:56.727+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:57.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:57.199+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:57.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:57.199+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:57.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:57.199+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:57.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:57.199+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:57.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:57.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:57.726+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:58.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:57 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:57.726+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:58.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:58.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:58.235+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:58.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:58.236+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:58.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:58.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:58.719+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:14:59.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:58 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:58.719+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:14:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:59.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:59.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:59.263+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:14:59.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:14:59.263+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:59.709+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:15:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:59.709+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:15:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:59.709+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:15:00.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:14:59 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:14:59.709+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:00.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:00.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:00.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:00.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:00.216+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:15:00.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:15:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:15:00.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:15:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:15:00.685+0000 7f7255350700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:19.502779+0000 (oldest deadline 2024-02-10T07:14:39.502779+0000) 2024-02-10T07:15:01.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:00 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1[54730]: 2024-02-10T07:15:00.685+0000 7f7255350700 -1 osd.1 47 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.24273.0:4420 3.19 3:9ae31074:::1000000462b.00000000:head [write 0~2420 [1@-1] in=2420b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:01.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:01.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:01.233+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:01.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:01 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:01.233+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:01.985 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:01 smithi073 podman[83865]: 2024-02-10 07:15:01.656476661 +0000 UTC m=+5.514556961 container died cffadaeafb44e8ea4d48e5cc1c81601b8661846573b234dedd89cee901f7f03c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1) 2024-02-10T07:15:02.240 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:01 smithi073 podman[83865]: 2024-02-10 07:15:01.985611458 +0000 UTC m=+5.843691815 container cleanup cffadaeafb44e8ea4d48e5cc1c81601b8661846573b234dedd89cee901f7f03c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:15:02.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:01 smithi073 bash[83865]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1 2024-02-10T07:15:02.241 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:02 smithi073 podman[83883]: 2024-02-10 07:15:02.195362707 +0000 UTC m=+0.537354812 container remove cffadaeafb44e8ea4d48e5cc1c81601b8661846573b234dedd89cee901f7f03c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1) 2024-02-10T07:15:02.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:02.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:02.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:02.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:02.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:02.240+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:02.603 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:02 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:02.240+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:02 smithi073 podman[83973]: 2024-02-10 07:15:02.506186462 +0000 UTC m=+0.017835356 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-10T07:15:02.871 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:02 smithi073 podman[83973]: 2024-02-10 07:15:02.65459291 +0000 UTC m=+0.166241795 container create de5705a399be898f30b8edafa26fd759369f2e91d249abbe4421db4fb3c38969 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:15:03.152 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:02 smithi073 podman[83973]: 2024-02-10 07:15:02.873945651 +0000 UTC m=+0.385594538 container init de5705a399be898f30b8edafa26fd759369f2e91d249abbe4421db4fb3c38969 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:15:03.153 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:02 smithi073 podman[83973]: 2024-02-10 07:15:02.877339576 +0000 UTC m=+0.388988469 container start de5705a399be898f30b8edafa26fd759369f2e91d249abbe4421db4fb3c38969 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T07:15:03.153 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:02 smithi073 podman[83973]: 2024-02-10 07:15:02.973846441 +0000 UTC m=+0.485495348 container attach de5705a399be898f30b8edafa26fd759369f2e91d249abbe4421db4fb3c38969 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T07:15:03.510 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:03 smithi073 podman[83973]: 2024-02-10 07:15:03.15306865 +0000 UTC m=+0.664717534 container died de5705a399be898f30b8edafa26fd759369f2e91d249abbe4421db4fb3c38969 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T07:15:03.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:03.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:03.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:03.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:03.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:03.238+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:03.511 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:03 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:03.239+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:04.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 systemd[1]: Stopping Ceph osd.2 for 3c2b87a2-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:15:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:03 smithi073 podman[83999]: 2024-02-10 07:15:03.794693922 +0000 UTC m=+0.631851754 container remove de5705a399be898f30b8edafa26fd759369f2e91d249abbe4421db4fb3c38969 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T07:15:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:03 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-10T07:15:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:03 smithi073 systemd[1]: Stopped Ceph osd.1 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T07:15:04.121 INFO:journalctl@ceph.osd.1.smithi073.stdout:Feb 10 07:15:03 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.1.service: Consumed 21.162s CPU time. 2024-02-10T07:15:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.216+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:04.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.216+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.247+0000 7fabe3625700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:15:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.247+0000 7fabe3625700 -1 osd.2 47 *** Got signal Terminated *** 2024-02-10T07:15:04.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:04 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:04.247+0000 7fabe3625700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:15:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:05.265+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:05.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:05.265+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:05.265+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:05.622 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:05 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:05.265+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:06.218+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:06.218+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:06.218+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:06.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:06 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:06.218+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:07.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:07.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:07.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:07.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:07.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:07.264+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:07.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:07 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:07.264+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:08.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:08.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:08.301+0000 7fabe1426700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.181:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T07:14:32.312144+0000 (oldest deadline 2024-02-10T07:14:52.312144+0000) 2024-02-10T07:15:08.621 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:08 smithi073 ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2[62076]: 2024-02-10T07:15:08.301+0000 7fabe1426700 -1 osd.2 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.24273.0:4416 3.1e 3:786b8d01:::10000004627.00000000:head [write 0~2445 [1@-1] in=2445b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T07:15:09.957 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:09 smithi073 podman[84077]: 2024-02-10 07:15:09.646728188 +0000 UTC m=+5.504118455 container died c93fd57a90db5bc5640d7b0b2da25ab2800343f961cb36f323156a5a70dda113 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:15:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84077]: 2024-02-10 07:15:10.055952198 +0000 UTC m=+5.913342462 container cleanup c93fd57a90db5bc5640d7b0b2da25ab2800343f961cb36f323156a5a70dda113 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T07:15:10.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 bash[84077]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2 2024-02-10T07:15:10.676 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84095]: 2024-02-10 07:15:10.390501373 +0000 UTC m=+0.737632540 container remove c93fd57a90db5bc5640d7b0b2da25ab2800343f961cb36f323156a5a70dda113 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:15:10.979 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84185]: 2024-02-10 07:15:10.676912067 +0000 UTC m=+0.110381762 container create 81f948b1aeda0d9fa681f83dd1506265830229064a73cc2ab1a0bfb1202525cc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:15:10.979 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84185]: 2024-02-10 07:15:10.584332851 +0000 UTC m=+0.017802514 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-10T07:15:10.979 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84185]: 2024-02-10 07:15:10.896233874 +0000 UTC m=+0.329703536 container init 81f948b1aeda0d9fa681f83dd1506265830229064a73cc2ab1a0bfb1202525cc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131) 2024-02-10T07:15:10.979 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84185]: 2024-02-10 07:15:10.899718162 +0000 UTC m=+0.333187815 container start 81f948b1aeda0d9fa681f83dd1506265830229064a73cc2ab1a0bfb1202525cc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:15:10.979 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:10 smithi073 podman[84185]: 2024-02-10 07:15:10.979321325 +0000 UTC m=+0.412790979 container attach 81f948b1aeda0d9fa681f83dd1506265830229064a73cc2ab1a0bfb1202525cc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1) 2024-02-10T07:15:11.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:11 smithi073 podman[84185]: 2024-02-10 07:15:11.178488643 +0000 UTC m=+0.611958303 container died 81f948b1aeda0d9fa681f83dd1506265830229064a73cc2ab1a0bfb1202525cc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131) 2024-02-10T07:15:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:11 smithi073 podman[84211]: 2024-02-10 07:15:11.724355916 +0000 UTC m=+0.540974433 container remove 81f948b1aeda0d9fa681f83dd1506265830229064a73cc2ab1a0bfb1202525cc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:15:12.121 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:11 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T07:15:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:11 smithi073 systemd[1]: Stopped Ceph osd.2 for 3c2b87a2-c7de-11ee-95b8-87774f69a715. 2024-02-10T07:15:12.122 INFO:journalctl@ceph.osd.2.smithi073.stdout:Feb 10 07:15:11 smithi073 systemd[1]: ceph-3c2b87a2-c7de-11ee-95b8-87774f69a715@osd.2.service: Consumed 19.187s CPU time. 2024-02-10T07:15:19.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:15:19.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:15:22.336 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:15:31.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:15:31.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:15:34.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:15:46.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:15:46.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:15:49.468 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:15:52.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:15:52.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:15:52.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:15:52.540 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:15:52.540 INFO:tasks.cephadm:Teardown complete 2024-02-10T07:15:52.541 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1591, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1848, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1724, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 180, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 348, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:15:52.563 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T07:15:52.574 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-10T07:15:52.591 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T07:15:52.591 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T07:15:52.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:15:52.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:15:55.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:04.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:04.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:07.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:19.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:19.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:22.816 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:25.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:25.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:25.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:25.884 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:16:25.927 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:16:25.945 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T07:16:25.957 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T07:16:25.957 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:16:25.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:25.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:25.980 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:16:25.984 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:16:25.984 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-02-10T07:16:25.984 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +810us[ +793us] +/- 107ms 2024-02-10T07:16:25.985 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -643us[ -659us] +/- 55ms 2024-02-10T07:16:25.985 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +408us[ +391us] +/- 66ms 2024-02-10T07:16:25.985 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:16:29.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:38.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:38.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:41.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:53.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:53.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:56.156 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:59.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:59.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:59.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:16:59.228 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:16:59.229 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:16:59.244 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T07:16:59.257 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T07:16:59.257 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T07:16:59.308 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T07:16:59.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:16:59.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:02.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:17:11.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:17:11.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:14.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:17:26.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:17:26.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:29.600 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:17:32.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:17:32.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:32.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:17:32.668 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:17:32.669 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:17:32.678 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T07:17:32.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T07:17:32.720 INFO:teuthology.task.internal:Duration was 3054.862794 seconds 2024-02-10T07:17:32.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T07:17:32.754 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T07:17:32.754 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T07:17:32.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:17:32.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:32.808 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:17:35.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:17:44.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:17:44.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:47.904 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:17:59.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:17:59.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:02.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:05.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:05.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:06.048 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:06.048 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:18:06.049 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:18:06.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T07:18:06.081 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T07:18:06.082 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T07:18:06.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:06.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:09.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:18.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:18.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:21.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:33.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:33.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:36.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:39.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:39.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:39.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:39.388 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:18:39.389 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:18:39.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T07:18:39.420 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:18:39.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:39.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:39.455 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2024-02-10T07:18:42.496 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:18:51.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:18:51.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:18:54.560 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:06.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:06.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:09.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:12.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:12.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:12.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:12.700 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:19:12.701 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:19:12.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T07:19:12.730 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T07:19:12.731 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144/remote/smithi073 2024-02-10T07:19:12.731 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T07:19:12.777 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144/remote/smithi181 2024-02-10T07:19:12.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:12.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:15.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:24.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:24.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:27.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:39.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:39.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:42.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:45.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:45.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:46.048 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:46.048 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:19:46.049 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:19:46.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T07:19:46.084 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T07:19:46.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T07:19:46.133 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T07:19:46.134 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T07:19:46.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:46.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:19:46.157 INFO:teuthology.orchestra.run.smithi073.stdout: 265582 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 10 07:15 /home/ubuntu/cephtest 2024-02-10T07:19:46.158 INFO:teuthology.orchestra.run.smithi073.stdout: 265583 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 10 07:18 /home/ubuntu/cephtest/archive 2024-02-10T07:19:46.158 INFO:teuthology.orchestra.run.smithi073.stdout: 265612 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:26 /home/ubuntu/cephtest/archive/syslog 2024-02-10T07:19:46.158 INFO:teuthology.orchestra.run.smithi073.stdout: 265670 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 10 06:26 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:19:46.158 INFO:teuthology.orchestra.run.smithi073.stdout: 265617 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 10 06:26 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:19:46.158 INFO:teuthology.orchestra.run.smithi073.stdout: 400386 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:16 /home/ubuntu/cephtest/archive/audit 2024-02-10T07:19:46.158 INFO:teuthology.orchestra.run.smithi073.stdout: 426276 176 -rw------- 1 ubuntu ubuntu 177077 Feb 10 07:16 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-10T07:19:46.159 INFO:teuthology.orchestra.run.smithi073.stdout: 265368 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 10 06:33 /home/ubuntu/cephtest/cephadm 2024-02-10T07:19:46.160 INFO:teuthology.orchestra.run.smithi073.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-10T07:19:49.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:19:58.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:19:58.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:20:01.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:20:13.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:20:13.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:20:16.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:20:19.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-10T07:20:19.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:20:19.424 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-10T07:20:19.424 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T07:20:19.425 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi181 2024-02-10T07:20:19.442 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T07:20:19.455 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T07:20:19.497 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-10T07:20:19.497 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T07:20:19.531 DEBUG:teuthology.nuke:shortname: smithi073 2024-02-10T07:20:19.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:20:19.561 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:18:12.058384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECQFzsaOFTh8GfRTSAuuSnfCajYZ6Z2X8NcWHGxHSyKXA+hxX9LMFhsdLST9AiQQVAF3Y7coUcH19fnM4j0OWg='} 2024-02-10T07:20:19.563 DEBUG:teuthology.nuke:shortname: smithi181 2024-02-10T07:20:19.565 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:20:19.590 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:18:12.060291', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFfrJwNnhIEcAMsqbv9JltU8sC0UhFdtsSv12aijf1VN69nMjVNqLoZw2zKY/IQdBcIxLXKRmXHyPAJbN+yqu0='} 2024-02-10T07:20:19.627 INFO:teuthology.orchestra.console.smithi073:Power off 2024-02-10T07:20:19.627 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T07:20:19.658 DEBUG:teuthology.orchestra.console.smithi073:power off output: Chassis Power Control: Down/Off 2024-02-10T07:20:19.658 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:19.688 INFO:teuthology.orchestra.console.smithi181:Power off 2024-02-10T07:20:19.688 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T07:20:19.702 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-02-10T07:20:19.715 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-02-10T07:20:19.716 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:19.753 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-02-10T07:20:23.704 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:23.753 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:23.823 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-02-10T07:20:23.887 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-02-10T07:20:27.828 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:27.888 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:27.939 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-02-10T07:20:28.000 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-02-10T07:20:28.101 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-02-10T07:20:31.940 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:20:32.052 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is off 2024-02-10T07:20:32.153 INFO:teuthology.orchestra.console.smithi073:Power off completed 2024-02-10T07:20:32.258 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3054.8627939224243 failure_reason: 'Command failed on smithi073 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c2b87a2-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8d65087d064143e1b5050021729893d2 status: fail success: false 2024-02-10T07:20:32.259 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T07:20:32.352 INFO:teuthology.run:FAIL